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ABSTBACT 

This report describes an evaluation of Project 
Inforaation Packages (PiPs># sets of manuals and other materials 
intended to help z. school district adopt and isplenent an exeaplary 
education project. Four PIPs were evaluated in a field test^ each PIP 
describing a different bilingual project. It was concluded that the 
awareness aaterials produced few applications for PIPs. Field-test 
sites that received PIPs tended not to. follow PIP guidelines closely^ 
but to adapt then extensively^ often vith good justification. The 
bilingual prograas'at the sites were collectively successful^ but the 
dissenination effort could Dot be judged a success. The present 
volume coaprises: (1) an executive suanary of the study 9uestions and 
findings, (2) an introduction to the study* (3i a non-technical 
suanary of the process evaluation cf the PIP diffusion efforts, and 
(H) a non-technical sunmary of the inpact evaluation of the diffusion 
effort on students. The voluae is intecded to provide a 
self-contained overview of the policy-related study guestions ar.d 
conclusions. (Aut hor) 
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This report is made pursuant to contract MOi 300--77--03l3i The aaount 
charged to the Departnent of Health, Education, and Welfare for the work 
resulting in this report (inclusive of the amounts so charged for any 
prior reports aubnitted under this contract) is $l,U3,205i The names 
of the persons, employed or retained by the contractor* vith managerial 
or professional responsibility for such vork^ or for tha content of the 
report* are as follows: 
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PREFACE 



This report describes an evaluation of Project Infornkation Packages 
(PIPs)» a specific type of packaging^ as field tested by the United States 
Office of Education (USOE) for the diffusion of four bilingual pro.jects* 
The field test began with the dissemination of the PIPs in the fall of 
1976* The evaluation described here began about nine months later (sunmier» 
1977) and continued through the 1978^1979 school year* 

This report consists of three volumes^ as follows: 

Volume I* the Present voluine * comprises (a) an executive sunanary of 
the study questions and findings^ (b) an introduction to the study (Sec* 
tion l)» (c) a non<*technical summary of Substudy I» the process evaluation 
of the PIP diffusion effort (Section 2)» and (d) a non*technical summary 
of Substudy II» the evaluation of the impact of the diffusion effort on 
students (Section 3)* This volume is intended to provide a self-contained 
overview of the policy^r elated study questions and conclusions* 

Volume 11^ the Technical Discui^sion and Appendices * documents the 
methodology and results of the two substudies and provides more detailed 
discussions of conclusions and recommendations* This volume also includes 
five appendices: (a) site-^by-dite results of the process substudy^ (b) 
site*by**site results of the Impact substudy^ (c) the complete conceptual 
framework used in the process evaluation substudy^ (d) a comparative analy* 
sis of the contents of the four bilin.gual PIPs* And (e) a summary of the 
major» mi^-study inputs from the study advisory panel* 

Vol^e III * is a collection of specific evaluation guidelines and job 
aides that were developed for the use of the field*test sites and which 
have been organized in the format of a Prototype Evaluation Manual * This 
volume should be viewed as a preliminary draft rather than a finished pro* 
duct. Further^ it deals in detail only with the evaluation of student 
achievement^ which is only one component of a complete^ bilingual program 
evaluation* 
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AK EVALUATION OF PEaiECT INFORMATION PACKAGES (PIPs) 
AS USED FOR THE DIFFUSION OF BILINGUAL PROJECTS 



EXECUTIVE SUMMARY 



The study described here was a field test of Project Information 
Packages (PlPs)i A PIP is a set of manuals and other materials intended 
to help a school district adopt and implement an exemplary educational 
projecti In this field test, four PlPa were evaluated, each of which 
described a different bilingual projecti 

This study wss one of a seriea of diffuaion'^ studies funded by the 
United States Office of Education (USOE) to investigate the effectiveness 
of PIP-type packagingi KMC has also participated in two other diffusion 
field tests in this serieSi These field testa evaluated PIPs that de- 
scribed compensatory^education projectSi The general concluaions presented 
here reflect the experience of all three field tests (see Bibliography, 
page 61) i 

The intended audience for this summary, and for the associated final 
report, includes those interested in the planning, implementation, or eval-* 
uation of large-scale, educational diffusion efforts # The study did not 
eicamine either the methods or the effectiveness of bilingual education t 

This summary is orgsnized under four headings: 

• Synopsis of the Bilingual-PIP Field Test 

• The Study Questions and the Dual RMC Role 

• Substudy I: Diffusion of Projects via PIPs 

• Substudy II: Impact on Achievement 

The first two sections provide the background for the study conclu* 
sions, while the latter two sections describe the major conclusions, 
recommendations, and products of the studyt 



"Diffusion** in this report means the transfer of educational proj-^ 
ects or practices to adopting school districtSi The term implies imple- 
mentation of the projects/practiceSi '^Dissemination,'* by contrast, means 
transmitting information about proJects/practiceSi Implementation is not 
necesrarily implied # 

2 

"Packaging,*' in this report, loeans (a) the systematic analysis of 

project features, plus (b) development of descriptive and how-*to-*do-*lt 
materials* 
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Synopsis of the Bllln^ual-^PIP Field Test 



Who Was Involved In the Field Test? 

The evaluation of the bilingual PIPs was funded by the Office of 
Education for a 30-month period (1977-1979), and conducted by KMC Research 
Corporation* The diffusion effort Involved 19 school districts across the 
country* each of which received an ESEA Title VII grant to Implement one of 
four packaged, bilingual projects* 



Whst Were the Origins of the Four Bllln&ual Projects? 

The four projects were originally developed b^ local school districts 
for their own students* were Identified as exemplary by another contractor 
In a USOE-spon sored nationwide search, and were validated by the Joint 
Dissemination Review Panel (JORF) * The projects are: 

# Project Adelante, from Corpus Chrlstl, Texas Spanleh/Engllsh 

# Project Nuevos Horlzoates, from Houston, Texas* ^Spanish/English 

# Project Savolr, from St* John Valley, Maine French/English 

# Project Venceremos, from Alice, Texas Spanish/English 

What Were the Orlglna and Contents of the PIPfi? 

The four PIPs were developed by CEMREL Inc* under a separate USOE 
contract* Each package consisted of a set of how-to*do-lt manuals, plus 
a synchronized tape and fllmstrlp, and some awareness materials* In 
general, a different manual was prepared for each type of project staff 
member— project director, teacher. Instructional consultant, evaluator, 
and so on* Some of the PIPs also Included a manual for the use of per- 
formance objectives! /l staff development ctanual, and a few site-developed 
Instructional materials* 

How Were the Blllnguel PIPs Disseminated? 

Dissemination and support services * The PIPs were disseminated by 
USOE via the network of IS Bilingual Training Resource Centers (TRCs) 
funded by the USOE Office of Bilingual Education (OBE)* These centers 
provided PIP-awareness materials to target LEAs In their regions and fol*- 
lovEd up with telephone calls* Many also helped LEAs prepare Title VII 
grant applications, and later provided staff training services under the 
same conditions that they provided training to other Title VII programs* 



The JDRP Is a panel formed jointly by USOE and the National Insti- 
tute of Education (NIE) to review all projects or practices proposed for 
dissemination under USOE or NIE sponsorship* 
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Grant application procedurea . LEAfi used PIP materials to help In 
preparing Title VII grant applications with the understanding that each 
successful applicant would receive a copy of the appropriate PIP at no 
cost. Bbveyer^ most applicants were Instructed not to mention the PiPs 
and were presumably Judged anonymously along with all other Title VXI 
applications. 



What Were the Results of the Bilingual PIP Diffusion Effort? 

Adoption . The awareness materials and telephone calls produced very 
few applications for PIPa. Host of the 19 adoptions resulted from per* 
aonal contacts between local personnel and state or federal officials. 

The process of matching the three different Spanish/English projects 
to specific adopter*8lte needs and resources was based on superficial 
project characteristics and» In effect » assignment was virtually random. 
The lack of procedures for selecting among the PIPs had little effect on 
the fleld-*test results* however* since there were few major differences 
among the PIP projects. 

Implementation . itHC dXte visitors reported that, In their Judgments^ 
the blllngual*program featuif^ implemented In most of the fleld^test sites 
ranged from adequate to excellent by current stsndatrda. However, since 
the focus was on the effectiveness of the PIP packages as diffusion tools, 
formal assessment of program quality was not Included In the study. The 
central question was whether the sites followed PIP guidelines closely. 

The answer waa that» In general, they did not. Fntenslve adaptation 
was the rule, often (In RMC's Judgment) with good Justification. In fact, 
the PIPs were found to be a relatively minor Influence Iji most sites. 
Program characteristics were shaped Isrgely by local factors, federal and 
state regulations, outside consultants, and neighboring LEA programs. 
However^ the lack of replication should noc be construed as a deficiency 
of the f leld-*test--slte programs. On balance, deviations from the PIP 
guidelines probably Improved suitability to local conditions* 

Thus, while the bilingual programs In the 19 fleld^test sites could 
be considered a collective success In terms of project features^ the PIP* 
based diffusion effort was not successful. The major breakdowns In the 
diffusion effort came In: 

• Establishing dlffuslon^system goals 

• Selecting projects for diffusion 

• Packaging the projects 

• Selecting and training diffusion agents 

In short, the major breakdowns occurred In the planning and preparation 
stages, well before any target LEAs became Involved In the diffusion 
effort. 
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The Study Queatlonfl and the Dual RMC Role 



There were two basic types of questioufi addressed in this study 
resulting, in effect, in two distinct substudieSi RMC played very 
different roles in the two substudiesi 



SubstudY It Diffusion of Projects via PIPs 

Substudy I question t The major focus of this substudy (and of the 
study as a whole) was on the effectiveness of the four bilingual*PIP 
p/ickages in establishing the exemplary projects in the field-*tedt siteSi 
This substudy was primarily a Process evaluation of project adoption and 
implementation, focusing on these four PIP Packages # It was not: (a) 
a study of the four bilingual projects chosen for PIP packaging, or (b) 
a comparison of alternstive diffusion approacheSi 

R MC role in Substudy I t RMC was an outside evaluator of the PIP 
packages i KMC did not participate in: (a) developing the four exemplary 
bilingual projects in the originating sites, (b) selecting the four proj-* 
ects for diffusion, (c) developing the four bilingual PIPs, (d) dissemi-* 
nating the four bilingual PIPs, or (e) implementing the projects in the 
19 field*test sites. 



Substudy lit Impact on Students 

Substudy II question t A secondary study question concerned the Im* 
pact of the PIP-based diffusion effort on stv^dent achievement (attitudes 
and other Impacts were examined, but in less depth)i That is: 

• Did program changes resulting from the diffusion effort lead 
directly to improved achievement? 

This question was of secondary interest, because there was no guar* 
antee that the intended program changes would occur and, in any caae, the 
programs could only be observed in their first two (developmental) yearSi 

Answering this question required a specialized, limited form of out* 
come evaluation t The study did not address the general outcome Question" 
^*How well did the bilingual-program students perform?" since change (Im-* 
provement) in student achievement is the goal of a diffusion effort, and 
high performance levels are not proof of improvement, nor are low levels 
proof that no improvraent has occurred # 

RMC role in Substudy li t RMC used local achievement evaluations from 
the 19 field*test sites in the attempt to determine the impact of the 
diffusion effort on student achievement # RMC also played a major role as 
consultant to the aites on outcome evaluation ! but final suthority was re- 
tained by the site, as was responsibility for all testing and for data 
analysis i 
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SubetudY I: Dlffuaion of Projecce via PIPb 



Conclufllonfl 

The principal contribution of the bilingual*PIP field teat was to 
reemphaslie the need for a more orderly, ayetematic approach to planninit 
all diffuaion efforte * This need was equally apparent in the two 
coapenfiatory^education^PIP field teatai 

The final report lor thia atudy treats a diffusion effort as a system 
composed of six malor elements : 

1* The diffuaion jgals set by the agency that plans the system* 
4 

2* The projects /practices available for diffusion, 

3, The target LEAa t with their specific attitudes* needs» and re- 
sources « 

4« The delivery system for bringing the projects /practices aad 
the target l£As together* including personnel* materials 
(e«g«» PIPs)» and all other resources* 

5i The Incentives for LEAs to (or not to) adopt projects and imple* 
ment them accurately! 

6* The competition faced by the system in the form of laws aad 
regulations* other formal or informal diffusion systems* etCi 

In the PIP diffusion attempts* these six elements were considered 
in isolation; some of them were not considered at all* The major re com* 
mendation from thia study is that all six elements should be considered 
explicitly and systematically , in planning or analyzing a diffusion study* 
The major product of the study is a conceptual framework that can facili-* 
tate such ayatematic planning or analysis # 

The following paragraphs first summarize the general conclusions of 
the three PIP field tests and then indicate the conclusions specific to 
thia bilingual*PIP atudy* The conclusions e:ce organized according to the 
six diffuaion^syatem elements listed abovei 



The report makes a critical diatinction between '^programs'* and 
^^projectSi" A "project" consists of procedures and guidelines* stsff 
specifications* materiala* and so on* that can* In principle* be trans<^ 
ported to an adopter aitei The project Is distinguished from the "context** 
which Includes the actual personnel* students* facilities* and community* 
A program ia a project as it actually operatea in s particular context 
(i*ei* program - project + context)i A major difficulty in many diffusion 
efforts is that a Program is validated but a project is disseminated * In 
aome cases* It appeara that the cont^t* not the project* is actually the 
exemplary component of the program and* of course* an exemplary cont^t 
(e*g** exemplary teachers) cannot be disseminatedi 

ix 
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t 1« Gosls of the planning agency «> At the most general level, the 

USOE gosl for the PIP diffusion efforts Was to promote cost«ef f ective im-* 
provement for students and staff* Specific PlP^dif fusion goals included: 

A* Diffusion of intact i exemplsry projects as validated by the JDRP* 

Rapid» efficient implementation in new adopter sites* 

C* Minimal technical sssistance (stand-alone packsging)* 

D* Positive impsct on student achievement. 

These goals may be reasonable ones for certain diffusion efforts 
involving structured* instructional procedures* and LEAs with strong 
incentives to follow the printed guidelines closely* However* as will 
be seen under the five following headings (i*e*» projects* target LEAs* 
delivery systm* incentives* and competition)* and under "Impact Conclu* 
sions*** these goals are not widely appropriate to and* specifically* are 
inconsistent with the realities of bilingual programs* The four goals 
listed above could be revised for bilingual^education diffusion efforts 
as follows: 

A* Diffusion of exemplary practices (i*e** components of bilingual 
projects) since most bilingual projects are too complex to be 
adopted intact by another school district (see footnote 4* above* 
and ^^projects/practices*" below)* 

B« Gradual imPlementation i with long*term staff development (see 
Target \EA3). 

C. A balsnce of technical assistance and Packaging j with packaging 
forma::s tailored to the needF of the LEAs* the characteristics 
of the exemplary practices* and the overall approach to technical 
assistance (see Delivery System)* 

D* Positive impact on student achievement should remain as one of 
the ultimate goals of bilingual-education diffusion efforts* 
but the problems of measuring impact in school settings should 
be recognissed. Practices should be validated under more con* 
trolled conditions than are possible in large-scale field tests* 
Field*test evaluations should focus on implementation of the 
exemplary practices (see Substudy II: Impact on Achievement)* 

2^ Projects/practices available for diffusion * It can be argued 
that the set of projects /practices is the most important element of the 
diffusion effort. The projects chosen for PlP-packaging were selected 
from among exemplary programs operating in school districts. Selection 
techniques did not separate context effects from project effects nor were 
project characteristics analysed adequately in terms of target-*LEA needs 
and resources. 
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# Prgjectfl or practlcea must be carefully analyzedt and selected 
to jat c h the needa and reaourcea of the target LEAs t For exam- 
ple, While the four bilingual projects choaen for PIP packaging 
depended heavily on good bilingual teacherc following general 
guidelinea, many field^teat aitea did not have auch teachera and 
therefore needed specific instructional procedurea that ^jould be 
uaed by monolingual Engliah^speaking teachers with Spanish /French** 
speaking aidesi 

# Adoption of whole, intsct projects is ressonable only for encap* 
sulsted, instructionsl projects (etgi, s pull«-out, resding lsb)i 
Projects sffecting the operstion of the school as s whole should 
be treated ss a collaction of components (management components, 
instructionsl components, psrent involvement components, and so 
on) that csn be disseminsted ss separate unitSi Typically, an 
exemplary bilingual program is made up of many such highly inter- 
dependent and context-determined components snd cannot be diffused 
as an intsct uniti 

3t Target LEAs * It is generally acknowledged that LEAs vsry widely 
in terms of: 

# Student needs 

# Staff and ctaterisl resources 

# Organizational structures 

# Resdiness and motivstion £or chsnge 

# Informstion seeking behavior * 

The PIP studies hsve shown thst these varistes affect both the kinds 
of projects/prsctices needed snd the kinds of disseminstion sctivities that 
are requiredi The nmqber of tsrget LEAa with similsr needs should slso be 
taken into account in planning diffusion activltiesi 

LEAs thst are potential tsrgets for the diffusion of bilinRusl proj- 
ects/practices cover the complete rsnges on sll of the sbove vsriste^i 
Therefore, the specific target pool must be carefully anslyzed snd clesrly 
defined before plsnning any bilingual*^proJect diffusion efforti Two im* 
portent generaliisstions csn be made: (a) Many target LEAs require exten- 
sive! long-term stsff development programs in order to implement successful 
bilingusl programsi Rapid start-up is simply not fessible; (b) Tsrget 
pools for Spsnish-English projects/prsctices are likely to be IsrgSi 
Target pools for other Isnguages will be much smaller and require very 
different diffusion spproschesi 

4t Delivery system # The delivery system includes all personnel, 
materials, snd other resources used to bring the projects /practices and 
the target LEAs togethert In this study, the msjor components of the 
delivery system were the PXPs and (secondarily) the TRCSi 




A delivery syst^ must provide awareness plus technical support for 
selection and implementation of the projects/practices * In most kinds of 
diffusion efforts » an integrated mix of personnel and packaging is needed 
for an effective delivery system* PIP-type» stand-»alone packaging is 
suitable under only a few» speciaiized conditions* 

The PIP studies have shown that printed materials (whether in PIP or 
other formats) are well suited to providing reference information* but not 
to; 

• focusing attention o;t key project festures, 
e persuading adopters to try new ideas, 

• identifying problems and providing feedback * or 

• providing reassurance , 

all of which Bre essential delivery^-system functions* Other media (eigi» 
film» videotape) appear to have some of the same probl^s» although they 
were not evaluated in these studies* 

The major recommendation of this study (and the other PIP studies) is 
that a systematic approach should be used in planning diffusion systems* 
In practice* this means that the delivery system must be designed to fit 
the goals* projects/practices* target LEAs, incentives* and competition 
that constitute the other msjor elements of the diffusion systemt 

In general, a combination of packaging and technical assistance will 
be required* However, elaborate packaging (e*g*, PIPs) is expensive and 
time consuming* It is cost effective only for Isrge pools of target LEAs, 
and for projects /practices that will not become obsolete too quickly* 

These ger.eralizations apply equally to delivery syst^s for bilingual 
or non^bilingual projects /practices* A need for complementary efforts 
in local capacity buildin g, while not unique to bilingual education, should 
be a special concern here due to the short aupply of experienced bilingual 
personnel in many target LEASi 

5* Incentives * The PIP studies and other diffusion studies have 
shown that ItEAs must be strongly motivated before they will make changesi 
Incentives to change may be provided by the desires of LEA personnel to 
iiq)rove programs or by legal requirements to provide new services* How- 
ever, these kinds of incentives usually do not motivate adopters to read 
printed materials carefully or to follow guidelines closelyt 

Certain external incentives«^specif ically the USOE requirement of de** 
tailed complisnce with guidelines as a condition of funding —proved effec- 
tive in the original, compensatory-education-PIP field test, but cf^atinuing 
use of funds in this manner is not consistent with current USOE p'^^ .cy* 

A reasonable expectation that target LEAs will be motivated to adopt 
the projects or practices should be a prerequisite for developing a diffu^ 
sion effort* Assuming that strong, externsl incentives ^o replicate 



xii 

? 0 



accurately will not be provided (and proba1>ly would not be appropriate), 
substantial adaptation of projects/practices (bilingual or otherwise) 
should be anticipated. 

6* Competition . A diffusion system intend&d to promote adoption of 
specific projects or practices nay be in direct competition with: (a) 
federal regulations or laws; (b) state regulations or laws} (c) local 
regulations, policies, and existing programs; (d) other LEAs promoting 
alternative practices; (e) professional organizations promoting altema* 
tive practices; (f) commercial organizations promoting alternative prac* 
tices; and (g) other diffusion systems promoting alternative practices* 

The alternative projects /practices and their sources must be under- 
stood if the diffusion of specific projects or practices is to be success- 
ful« For each potentially comp^^ting source, the diffusion-system planners 
must decide whether to adapt to the conflicting position (often required 
where laws are involved) or to compete directly* This principle holds in 
all areas of education, but bilingual education is affected by an espe* 
cially wide variety of laws and regulations that must be taken into ac- 
count in planning a diffusion effort* 



Summary of Substudy I Conclusions 

The major conclusion of this and the other PIP studies is that a 
successful diffusion effort ^taquires systematic planning * This planning 
must take into sccount: 



• 


goals 


• 


projects /practices 


• 


target LEAs 


• 


delivery syst«D 


• 


incentives 


• 


competition 



Substudy I Products 

The major product of the diffusion substudy is a conceptual frame* 
work that can facilitate planning (or analyzing) a diffusion system* The 
conventions that were developed for generating such frameworks are de- 
scribed in Volume II of this report* The specific framework that was 
used in the Substudy I analysis is included as an appendix in Volume II* 

A second product is a detailed, comparative analysis of the features 
of the four projects described in the PIPs. This analysis is also included 
as an appendix in Volume II, and is intended for the reader who wishes (a) 
to determine whether the PIPs contain information relevant to his or her 
interests, or (b) to understand more clearly why KMC Judged that there 
were few real differences among the four bilingusl projects* 
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Substudy II; Impact on Achievement 



Conclusions 

In the impact substudy » KMC consulted extensively with the field* 
test sites on the design and Implementation of their required* Title VII 
project evaluations^ RMC then reviewed (and» In some cases» reanalyzed) 
the local data» thoroughly and attempted to use the results to determine 
the impact of the diffusion effort on student achievement (aee page 4)* 

This attempt was unsuccessful. While subjective impressions aug* 
gested that some (perhaps most) of the 19 programs were effective* technl-* 
cal problems with the evaluations precluded any clear* objective evidence * 
positive or negatlve t on the extent to which the diffusion effort jjfected 
student performance ^ 

Furthermore* these technical problems are not specific to the 19 
fleld*test sites or to bilingual programs^ Although some of these prob* 
laiks make blllngual-*program evaluation especially difficult* they effect 
all program ImPact evaluations i at least to some degree^ The major prob* 
lemB are listed below^ 



Problems Beyond the Control of Evaluators 

Lack of suitable comparison standards ^ Detection of program Impact 
would require ongoing maintenance of extenalve* extremely precise* base-* 
line data for all district studenta. Few* If any* school districts main* 
tain data bases that are adequate for detecting program Impacts^ Norm* 
referenced comparisons are not sufficiently precise for this purpose. 
Control groups are generally unavailable for bilingual or compensatory 
programs* since such programa usually serve all students who meet local 
selection criteria* 

Lack of adequate tests . Standardized teats In English may be satis- 
factory for use In some impact studies* although they are not always suit- 
able In bilingual programs^ Few achievement tests are available In lan- 
guages other than English* and completely satlsfsctory tests of language 
proficiency have not been developed^ 



Other C^ T pn" Technical Problems In Program Evaluations 

Lack of student descripti on. Many evaluations fall to group students 
for analysis according to their skills and other characteristics^ In bi- 
lingual programs* where some students speak fluent English and others very 
little or none at all* such grouping Is especially critical. 

Lack of program description . Many evaluations fall to group students 
according to the Instruction they receive. This is essential In all bllln* 
gual* compensatory* or other Individualized programs In which the Instruc*' 
tlon may vary widely from student to student. 
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Lack of match between test content sod curriculum ^ The exact degree 
to which tests should match the curriculum is sn importsnt, unresolved 
issue in evaluation* However, in msny evslustions, the tests besr slmost 
no relation to the subject matter being tsught, snd are obviously insensi- 
tive to progrsm effects* 

Use of incorrect test levels * Test levels that are too essy or too 
difficult cannot measure the true impact of a program* Determining the 
correct test level is often sn important, practical problem* 

Inappropriste testing and scoring procedures * Testing and scoring 
procedures present no real technical probl^s but, in practice, inadequate 
procedures sre common sources of evsluation errors* 

Insppropriste analyses * This concern is not with the unresolved, 
sophisticated issues in data analysis, but rati ^r thst many evaluation 
reports still include generslly discredited an .yses (e*g*, posttest-* 
pretest gains, and grade**equivalent gsins) (Horst, Tallmadge, and Wood> 
1975)* 

Lsck of interpretation * Many evaluators report sny apparent increase 
in test scores as s positive result, with no attempt to show (a) that the 
gains are greater than would be expected from the same students if they 
were not in the program, or (b) thst progrsm activities could hsve been 
responsible for the incressed gains* 



Why Do These Evaluation Problems Persist? 

The above evaluation flaws are not restricted to bilingual^program 
evaluations* In fact they are slmost universsl, even (as in the PIP study) 
when evaluations are conducted by highly competent evaluators* Only s few 
of the f ield^test'Site evsluators were able to produce above^average eval* 
uation reports* Some major reasons for this situation appear to be: 

♦ Lack of valuator time * Many evaluators are funded for a few 
days to complete tasks that require weeks* 

♦ Conflicting and impossible demsnds * Local, state, snd federal 
evaluation guidelines and requirments are often technically 
inappropriate and may conflict with each other* 

♦ Uncritical audience .. Evsluators often believe that their reports 
will not be read by any technically qualified persons, or perhaps 
not read by anyone at all* 

In short, the technically inadequate local report appears to be the result 
of assigning the evaluator an impossible and futile task. 



• Impl^entation of project features should be confiraed before 
major Impact studies are undertaken* 

• Impact studies are not fessible in many local school districts 
and should not be required unless appropriate conditions are 
knovn to exist and adequate resources are provided « 

• From the two recommendations, above, it follovs that diffusion* 
system field tests should use process-evaluation methods * Eval- 
uation of Impacts on students should be restricted to a few, 
carefully selected sites where iiq)lementation has occurred and 
the technical requirements for Impact evaluation can be met* 

• Appropriate evaluation guidelines should be provided to local 
evaluators* 



Substudy II Product 

The major product of the impact substudy is a draft evaluation manual 
for bilingual programs* The draft manual is included as a separately bound 
appendix to the final report (Volume III)* It consists essentially of the 
recommendations developed by KMC for the field-test sites, organized into 
a single volume* The manual deals primarily with student achievement out* * 
comes * Brief sections daal with other Important evaluation topics (e*g*» 
formative evaluation, student affect, staff development, and parent/conmun- 
ity involvement) but the manual is incomplete in these areas* The manual 
is in no sense a finished product but, rather, a starting point intended 
to spotli^t some of the unresolved technical and philosophical issues in 
bilingual program evaluation* 
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It INTRODOCTION 



It 1 The Vocflbtilflrv of the Report 

The fields of educational diffusion and bilingual education are re* 
plete vith special definitions and abbrevlationa^ and there are several 
basic definitions and distinctions that must be clarified before any dia* 
cussion of the study ia poasiblet Definitions refer only to usage in this 
report^ and no assertions are made as to their applicability in other con^ 
texts t 

Itltl niQftrnn^firttiQn TermlnQloflY 

Itltltl Dissemination vst Diffuaion 

Dissemination : Following the well establlBhed usage in com* 
munication theory^ 'Mis semination** is used to mean the distri- 
bution of information or of materials containing informationt 
ThuSy PIPs have been disseminated when they have beer distrib- 
uted to LEASt Implementation ia not necessarily impliedt 

Diffusion I This term ia used to mean the transfer of practices 
to new audienceSt Thus» a project has been diffused when it is 
actually opferating in adopter siteSt In general^ dissemination 
of information is one of the processes used to diffuse exemplary 
projects and practiceSt 

ltltlt2 Packaging vst Project Information Packages (PIPs) 

Packaging : This term 1b Intended to Imply (a) careful analysis 
of the exemplary projects and (b) development of descriptive and 
"how-to'do'it" materials t 

Project Information Package (PlP) t A PIP, as the term is used in 
this report, implies a apecialized form of Package that incorpo* 
rates three basic concepts: 
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• Whole^Project adoption ) the valldatloot packaging, and dif- 
fusion of completei exenpltry projecta as intact units, rather 
than the diffusion of selected parti of a project* 

• Rapid start"up ) capitalizing on the experience of the de-- 
veloper site to drastically reduce the time required by an 
adopter site to reach full operation* 

e Stand-alone packages ) the use of packaging as the primary 
means of diffusion « with a minimum of technical assistance* 
This implies a much more elaborate set of materials than 
would conventional diffusion methods that provide a great 
deal of technical assistance* 

As the term "PIP*' Is used In this report, an LEA cannot "Imple-* 
ment" a PIP* The PIP Is a tool that Is used In implementing 
a project* It Is the project described In the PIP that Is 
Implemented* 

1*1*1*3 Project, Context* and Program 

Project ) Specific procedures* staff specif lea t Ions « guldellnesi 
and materials that can* In principle, be transported to an adopter 
site* A project, as the term Is used In this report, can be 
described without reference to psrtlculer Indlvldusls, or loca- 
tions * 

Context * The particular staff members i students, schools, com- 
munity, and so on that are part of a program but cannot be dis- 
seminated to adopter sites* 

Prog ram i A ^'program*' Is a combination of a "project** plus a 
"context**' That let 

Program Project + Context 



This is a critical distinct ion t because t intentionally or nott 
it is uaually a program that is validated, and it is always pos* 
sible that the developer^sitft context (etgt, an exceptional 
staff, that cannot be disseminated) is largely responsible for 
achieving exemplary atatus. The project might consist entirely 
of conventional practices that are not unusually effective except 
in the hands of exceptional personnelt 

ltltlt4 Adoption vs^ Implementation 

Adoption : The decision or commitment by an lEA to Implement a 
project , 

Implementation : The actual preparation for and operation of a 
project t 

These terms are used in a general sense in this report and should 
not be confused with the terms, "Selection/Adoption,** **Start*up," 
and '^Operation*' (used in Volume II) ^ich refer to stages of the 
PIP*Dif fusion Model, 

1,1,1,5 Replication vs. Adaptation 

Replication (of a PIP project by an sdopter site): (a) Establish*- 
ing specific procedures to ensure that all PIP-specif ied project 
goals are met i and (b) using the procedures described in the PIP 
except for minor instances where differences in contexts between 
developer and adopter sites require that procedures be changed 
in order to achieve PIP-specif ied goals. 

Adaptation : Changing project goals or procedures to suit adopter* 
site characteristics or the preferences of the adopter-site per- 
sonnel, ^^Adsptatlon'* and "replication** are not mutually exclu* 
sive terms and the decision as to whether a site has replicated 
or adapted a project is highly subjective. 



2 Blllngual*EducatloD Temlnolpgy uaed In this Report 



1.1.2.1 BlllDRual Program va. BlllpRual Project 

Bilingual program : A program of instruction ualng two languagas 
and uaually involving several years of achoollng and several 
aubject areas. Title VII bilingual programa in elementary 
schools are Intended to provide Instruction In and atudy of 
Engllah» plua Inatructlon in the native language (to the extent 
neceaaary to progreaa through achool) and» In general* they also 
Include staff development » culture and heritage* and community 
Involvement coaponents. 

Features of bilingual programa differ, and the exact definition 
Is not critical In thla report, but It la important to note the 
distinction between a bilingual program , which includes the 
^'context,'* and a bilingual project . Which does not. 

Bilingual project ; A apeclflc set of guidelines, procedures, 
materials, and ao on for operating a bilingual program (aee 
the definition of "project, under Dissemination Terminology, 
above). 

1.1.2-2 Limited Engll8h"8peaklng fLES> : 

This term is used in only a few Isolated places in the body of 
thla report, and la redefined in each Instance. This policy was 
adopted because of the many different interprets tlona placed on 
the Title VII definition of LES and the wide variety of criteria 
(performance on various teats, teacher judgment, home language) 
used in dealgnatlng students aa LES. ^^LES" la used in Appendix A, 
the site-by* site summary of diffusion results, aa the term was 
used by each field-test alte. Deflnltlona are not necessarily 
consistent from site to site. Although the term has now been 
largely replaced by lEP (limited English proficiency), LES was 
more generally in uae at the beginning of the study. 



1.1.3 Abbreviations 



ASK Analysis and Selection Kit 

BESC Bilingual Educational Service Center 

CODOFIL Council for the Developtoent of French in Louiaiana 

CL community liaison 

DAC Dissemination/Assessment Center (now EDAC) 

DFP director of federal programs 

DH decision msker 

DOE Department of Education 

EV evaluator 

FES fluent English-speaking 

IC instructional consultant 

JDRP Joint Dissemination Review Panel 

LEA local educational agency 

LEP limited English proficiency 

LES limited English-^speaking 

WjC Materials Development Center 

NES non-Engliah-s peaking 

OBE Office of Bilingual Education 

OCK Office of Civil Rights 

OED Office of Evaluation and Dissemination (formerly OPBE) 

OPBE Office of Planning^ Budgeting and Evaluation (now OED) 

PAC parent advisory committee 

P/C parent/ community 

PD project director 

PIP Project Information Package 

PS project secretary 

SDC staff development consultant 

SEA state educational agency 

T teacher 

TA teacher aide 

TRC Training Resource Center (now BESC) 

USOE United States Office of Education 
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It2 SYnoPala of Elllngual-'PIP Field Teat 



lt2tl Who Waa Inyolvcd In the pield Test? 

The evaluation of bilingual project lapletaentar-t-on via Project Infor* 
matlon Packages (PIPe) vaa funded by the Office of Education (USOE) for a 
30-month period ( 1977-1979) t and conducted by BMC Research Corporation* 
The study waa designed to determine the ef f ectlveneaa of PIPs In helping 
to diffuse exemplary bilingual projecta* 

The diffusion effort Involved 19 achool dlatrlcta, each of which 
received an ESEA Title VII grant to Implement one of four packaged bllln* 
gual projectat 

l,2t2 What Were the Orlglna of the Four Bilingual ProJecta? 

The four projects were originally developed by local school dlstrlcta 
for their own atudents, were Identified as exemplary by the American Inatl*- 
tutea for Research (AIR) In a USOE^spoitEiored nationwide aearch^ and were 
validated by the Joint Dlaaemlnatlon Review Panel (JDRP), comprising USOE 
and National Inatltute of Education (NIE) personnel* The projects were 
Identified on the baala of apparent effectlveneaa In Improving atuditnt 
achievement* None Incorporated features Judged difficult to duplicate, 
except for the requirement of experienced bilingual ataff* The projects 
are; 

• Project Adelante* from Corpus Chrletl, Texaa Spanlah /English 

• Project Nuevoa Uorlzontea, f rom Houaton* Texaa* «* *Spanlsh/Engllsh 

• Project Savolr, from St* John Valley, Maine French /Engllah 

• Project Vencerenoa, from Alice, Texas Spanlah/Engllsh 

1*::*3 What Was Included In the Four Bilingual PlPa? 

The PlPa, developed by CEHREL* Inc* under a aeparate USOE contract^ 
each conalsted of a aet of how<-to«do«lt manuals plua «: aynchronlsed tape 
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and filmstrip, and some awareness materials* In general, a different 
manual was prepared for each type of project staff member: project di* 
rector, teacher, instructional consultant, evaluator, and so on. Some of 
the PIPs also included a manual for the use of performance objectives, a 
staff development manual, and site*developed materials* 

Since staff positions differed from one project to the next, the sets 
of manuala also differed from PIP to PIP. In all the PIPs, however, the 
key manual was the Project Management Director:^ , which was designed to be 
used by the project director* A typical Project Management Directory was 
about 175 pages long with the following table of contents: 



Chapter 


1 


Project Overview 


Chapter 


2 


Using the PIP 


Chapter 


3 


Management Approach 


Chapter 


4 


Communicating with School and Community 


Chapter 


S: Continuing Beyond the First Year 


Chapter 


6 


Budget 


Chapter 


7 


Selecting Students 


Chapter 


8 


Classroom Implementation 


Chapter 


9 


Selecting Staff 


Chapter 


10 


Staff Development 


Chapter 


11 


Staff Relationships 


Chapter 


12 


Materials/Equipment 


Chapter 


13 


Facilities 


Chapter 


14 


Goals 


Chapter 


15 


Task Checklists 



Each chapter listed the relevant project goals and the associated 
tasks for the project director, then provided a narrative discuasion of 
the topic, illustrated with occasional chares or forms from the developer 
site. Appendix C provides a detailed analysis of the contents of the four 
bilingual PIPs. 
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I,2i4 How Were the Bilingual PlPa LJ^aaemlnated? 



Dlaeemlnatlon and support aervlces t The basic dissemination system 
for the bilingual FIPs wae the network of IS Bilingual Training Resource 
Centers (TRCfi) funded by OBG* Under the direction of USOG» these centers 
provided PIP-awareness materials to target LEAs In their regions and fol*- 
lowed up with phone calls* Hany also helped LEAs prepare Title VII grant 
applications, and later provided staff training services under much the 
same condltlcmi that they provided trelnlng to other Title VII programs* 

Grant application Procedures * LEAs used PIP matfirlals to help In 
preparing Title VII grant applications with the understanding that each 
successful applicant would receive a copy of the appropriate PIP at no 
costi However, most applicants were Instructed not to mention the PIPs 
In their applications, and were presumably Judged anonymously along with 
all other Title VII appllcstlonSi 

1.^,5 What Were the Results of the Blllnftual^PIP Diffusion Effort? 

Adoption * The brochures and phone calls produced very few applica- 
tions for PIPs* Host of the 19 adoptions £«sulted from personal contacts 
between local personnel and state or federal oifflclals* 

The process of matching the three different Spanish /English projects 
to specific adopter-site needs and resources was based on superficial 
project characteristics and. In effect, assignment was virtually random* 
However^ because of the similarity among the P7? projects, this was not 
a major problem In the field test* 

Implementation * BMC site visitors reported that. In their Judgments, 
the bilingual program features Implemented In most of the fleld*test sites 
ranged from adequate to excellent by current standards* However, since 
the focus was on the effectiveness of the PIP packaRes ss diffusion tools, 
rather than on the bilingual programs of the study, no formal assessment 
of program quality was Included In the study* The central Issue was how 
closely the sites followed the PIP guidelines* 
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The answer was that, in general* PIP guidelines were not followed In 
datailt Extensive adaptation was the rule, often (in HHC's Judgment) with 
good Justification* In fsct, the PIPs were found to be relatively minor 
influences in most sites* Program characteristics were shaped largely by 
local factors, federal and state regulations, outside consultants, and 
neighboring LEA programs* 

Thus, while the bilingual programs in the 19 field^test sites could 
be considered a collective success in terms of project features, the PIP<- 
based diffusion effort was not successful* The major breakdowns In the 
diffusion effort came in: 

• Establishing dif fusion- syet em goals 

• Selecting projects for diffusion 

• Packaging the projects 

• Selecting and training diffusion agents 

In short, the major breakdowns occurred in the planning and preparation 
of the diffusion effort, well before any target LEAs became Involved in 
the field test* 
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It3 An Qvcrvletf of the Seven USQE PIP Studies 

1*3,1 Where Did This Study Fit Into the Broader USPS Packaging Investl" 
gat Ion? 

This was the final study In a set of seven. Together* the seven 
studies represent a slx^year Imrestlgatlon by USOS Into PIP*tvpe packaging 
as an approach to the diffusion of exemplary projects* The seven studies 
Involved three sets of projects* all developed In local school dlstrlcta» 
and four sets of PIPs» developed by two different contractors: 

Projects PlPa 

A, 6 Compensatory Education Projecta * First Wave a« Prototype PIFa 

b« Revised PIPs 

B, 6 Compensatory Education ^rojecta - Second Wave c* Flrst^draft PlPa 

C, 4 Bilingual Projects d* Flrat*draft PlPa 



Vv<m RMC's perspective* the etudles were conducted In two» overlapping stsges: 



StaRe I: Prototype Development and Trvout (Years 1-3) 



1* 6 First-wave Comp*-Ed Projects 

2, 6 Flrst^ave Ccmp-Ed Projects 

3, 4 Bilingual Projects 



Identify projects BMC 
Develop prototype PiPe 

FIELD TEST prototype PlPa SRi/BMC 

Identify projects AIR 



Sta&e II: Revisloni Expanalon, and Trvout (Years 3-6) 

4, 6 Flrst*Wave Conp-Ed Projects Revise prototype PlPa 

5, 6 Second*Wave Comp-Ed Projecta Identify projects 

Develop Comp^Ed PIPs 
4 Bilingual Projects Develop Bilingual PIPs 



6, 12 Comp-Ed Projects 
7* 4 Bilingual Projects 



FIELD TEST all Comp-Ed PIPs 
FIELD TEST Bilingual PIPs 



RHC 
CEHREL 

AIR/RMC 
RMC 



Study 7 Is the subject of this report s Studies 1* 2^ 4* 6» and 7 
were closely coordinated and shared common RMC staff « Studies 6 and 7» 
the Stage II field test&» overlapped and should be viewed as psrallel 
rather than sequential studies. Studies 3 and S were Indirectly coordi- 
nated with the others through USOE (see Figure 1), 
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STAGE I STAGE II 






73-74 


74-75 1 


75-76 


76-77 1 77-78 1 


7fl-V9 1 




(1) 


(2) 










Identify 












6 Prototype 


Projects 


Field*Test Study of 








Comp«Ed 


Develop 6 


6 Prototype Co«p*Ed PIPs 








PIP8 


Comp^Ed 














PIPS KMC 




SKI/RMC 














(4) 


(6) 










Redesign 6 








6 Redesigned 




Prototype 








Comp-Ed PIPs 




Comp-Ed 


Field*Test Study of 










PIPs 


6 Redesigned plus 6 Nev 










KMC 


Comp-Ed PIPs 










(5) 














Identify 








6 Nev Comp-Ed PIPs 




Projects 


(Dissemination by USOE) 










Develop 6 




air/khc 










Nev Comp*Ed 












PIPs 












(3) 






(7) 


4 Bilingual PIPs 


Identify 


Develop 


Disseminate 


Bilingual PIP 






4 Bilingual 


4 Bilingual 


4 Bilingual 


Field-Test Study 






Projects 


PIPs 


PIPs 


(This Study) 






AIR 


CEHREL 


USOE 




KMC 



Each Row represents a different set of PIPs * The first two rows represent the same .Projects 
but different PIPs . 

Each .Bo x represents one contract * Drjted lines between boxes 2 and 4» and boxes 6 and 7 
indicate closely coordinated studies, with overlapping staffs* 



Figure I* Summary of the USOE PlP^packaging investigation. 



It3t2 What Were the Reaulta of the Other PIP Fleld'Tcats C&tudlea 2 and 6)? 



The following actlvltlea all coincided with the field test of the pro*- 
totype, compenaatoryeducatlon PlPa (Study 2)t (a) Identification of the 
bilingual projecta and second-wave compenaatory-educatlon projecta; (b) the 
development of the correapondlng PlPa; and (c) planning of the later field 
testa (Studlea 6 and 7)* Thus, the bilingual PlPa benefited only partially 
from the reaults of the flrat field teat (Study 2)* The field teit of the 
twelve, compenaatoryeducatlon PlPa (Study 6), coincided with the bilingual-* 
PIP field teat (dlaaemlnatlon and Implementation stagea) and, therefore, the 
experience gained In Study 6 did not have a major Impact on the bilingual* 
PIP dlffualon effort. 

The conclusions sunanarlzed here reflect the overlap among all the 
studlea and. In particular, the fact that Study 6 and this study, the 
blllngual'PIP'f leld*teat atudy (Study 7), overlapped substantially In time 
aa well as staff , 



Study 2i Field test of 6 Prototype* Compens a to rjf- Education piPg 

Nineteen LCAs were given two-^ear (1974*1976) ESCA Title III grants 
(ranging from about $50,000 to $250,000 for the first year) contingent 
upon following PIP guidelines and cooperating with the fleld*test study. 

Conclusion : A PIP*bssed dlf fusion -SYst em can lead to accurate rePllca* 



tlon If: 



e Projects match LEA needa and resources 
e Strong monetary Incentives are provided 




Study 6: Field Test of U Com&flnfiAtory-Education PIPs 

Year I 

PIPs were disseminated by eight regional contractors who provided pri* 
marily awareness activities, PIP materials were free» but LEAs were 
expected to use local; state* or federal funds obtained through conven*^ 
tional procedures (not contingent upon replication of the PIP projects). 

Conclusion : This PlP^basea diffusion system led to very few accurate 
replications because : 

# PIP projects were adopted with little regard for LEA 
needs and resources 

# LEAs had little Inclination to follow PIP guidelines 

Year 2 

PIPs were disseminated via the Rational Diffusion Ketwork (HDK), The 
develoPer*site Project director for each PIP Project was funded as a 
DeveloPer/Pemonstrator (DP) to provide awareness and technical assis-* 
tance services, NDK State Facilitators provided additional awareness 
and coordination services. 

Conclusion : This diffusion system was more DD*based than PIP*based, 
Some DDs made major use of the PIP materials. Others 
did not. Effectiveness varied » depending on the projects 
and the DDs, Accurate replication occurred only when: 

# DDs ensured a good match between the projects and LEA 
needs and resources 

# DDs provided the incentives to replicate in the form 
of personal contact and persuasion. 
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1,4 The Dual Focua of the PIP Field Testa: 



"Prolect Dlffuftlon'* Veraua '^ImDact op Stydgnta'* 



While there are ntany reaeona for dlffualng exemplary projects or 
practices^ one of the major goals Is to Improve student achlevementi 
The desired result may be represented as: 




Diffusion 
System for 
bcemplary 
Practices 




[Link 
Diffusion 



Correapondlng 
Changes 
in LEA 
Pr< 



Link 
Impact 




Improved 
Student 
Achievement 



Figure 2i Intended dlf fualon*^ayatem reaultSi 



Since the purpose of s diffusion system Is to produce change* Improyemftnt 
Is the criterion of success snd» Ideally, one would like to evsluate Link 1 
In the dlsgram directly: 



• Link 1: Does the diffusion system lead to Improved student schleve- 
ment? 



In a field test» however^ there sre too many vsrlsbles Involved to 
snstfer this question dlrectlyt If diffusion does not lead to improved 
schlevement, such sn evsluatlon provides no^ Information ss to why, or whst 
to do nexti Therefore^ ss s prsctlcsl matter^ It Is necssssry to examine 
Links 2 snd 3 sepsrstely. 



• Link 2: (Diffusion Question ): Did the diffusion system lesd to the 
intended LF.' progrsm chsngea ^prAffl^ry Btudy question)? 



• Link 3: (Impsct Question ) i Did the ohsnges In the LKA programs 

(whether related to the diffusion effort or not) lesd to 
Improved student achievement ( secondsrv study question)? 
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In order to understand the impacts on students, we must, of course, 
first understand the effects on the schools* If, for example, it were 
found that the diffusion efforts were unsuccessful in establishing the 
intended projects in the schools, it would be appropriate to recommend 
changes to the delivery system in order to improve implementation of the 
projects* If it were found that the projects were Implemented very well, 
but that there were no positive effect on the students, then it would be 
appropriate to recommend changes in tha projects * 

In principle, it does not make sense to ask the impact question until 
it is known that implementation has occurred* In practice, hovever, it 
is not always reasonable to wait for the snswer to the diffusion question 
before beginning the Impact investigation* The important point to empha- 
size is that, in thia study, the attempt to evaluate project impact was 
independent of how well the projects, as described in the PlPa, had been 
Implemented* The attempt was to evaluste each project as it was operating* 
Interpretation of project Impact, of course, must depend on the features 
of the project that are in place; but, because the two questions had to be 
addressed concurrently, the study approach was to develop the best possible 
outcome evaluation for each site, whether or not the PIPs had been effec- 
tive in producing faithful adoptions* 

In practice, there is often no clear, simple answer to either ques*" 
tion* However, the division into diffusion questions and impact questions 
provides an understanding of the diffusion problems and the policy impli* 
cations* Therefore, diffusion and impact are treated as two distinct 
substudies throughout this report and, correspondingly, the body of the 
report is organized into two sections: 

Substudy I: Diffusion of the Projects (Process Evaluation) 

Substudy II: Impact on Students (Outcome Evaluation) 

In general, ^^impact," as used in this report, refers to impact on 
student achievement * However, affective impacts are also considered 
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briefly! aa are Impacta of parent/conuaualty componenta of th'^ bilingual 
programa. Impacta of ataf f^development componenta are dlacuaaed uader 
both Subatudlea I aad II although, atrlctly apeafclng, ataff development 
la a part of project implementation rather thaa a part of the ultimate 
program outcomea. 



2. SUMMARY OF SUBSTUDY I; PROJECT DIFFUSION* 
2i 1 Substudy I Scope and Results 

21111 What Were the Diffusion Substudy Questions? 

The bilingual*PIF diffusion study was designed to add to the general 
understanding of diffusion systems as well as to evaluate the usefulness 
of the four bilingual PIPsi The specific questions weref 

# Adoption 

~ What Influenced potential adopter LEAs to adopt or not to 
adopt the bilingual<^PIP projects? 

— To what extent were the projects that were chosen by adopters 
appropriate to those adopters? 

• Implementation 

» What fsctors (PIP» context* other) influenced implementation? 

— What were the characteristics of the resulting projects* and 
how closely did they resemble those described in the PIPs? 

21112 What Was the KMC flole in the Diffusion Substudy? 

RHC was not responsible for the development or dissemination of the 
bilingual PIPsi The RMC role in the diffusion substudy was that of a non* 
reactive observer of the PIP*dif fusion field test- Unlike the consulting 
role in the impact substudy* RHC was not intended to have any traininftt 
consultinfti or feedback role in the diffusion substudy, as this would have 
invalidated the field**test findingSi 

2.1i3 Wh'' Was the Research Approach to Answering the Diffusion Questions? 

2iU3-jl Pro cess -evaluation spproach t In the siiqilest terms* the 
diffusion substudy was based on a process evaluation that consisted of (a) 
describing the complete diffusion system as it was intended to operate* 
(b) describing the system aa it actually operated, (c) comparing the two 
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dedcriptions and analyzing di6crepancied» and (d) proposing changes to 
correct the problemSt Recommendations, imder this approach^ could include 
either changes to the system goals, or changes to the procedurfi for meet-* 
ing the existing goals* 

2tlt3t2 Dif fusion^system modelinft t In principle, this approach i« 
little more than common senset In practice, however, the many people and 
organizations involved in a diffusion aystem, and their wide variety of 
goals, procedures, materials, and so on, make it difficult to describe 
such a system in a way that captures the roles of all the parts of the 
system and displays their interactions clearly* The key to the process 
evaluation described here is an approach developed by KMC in Studies 2 
and 6 for the concise, accurate description of diffusion systems* The 
descriptions, or "models^^ are abstractions of the actual and intended 
diffusion systems, and include only those features that are relevant to 
understsnding and revising the systena* 

2*1*3*3 Data collection * Data on dissemination and implementation 
of the projects were collected by four two^person teams using guides for 
unstructured interviews and classroom observation* Due to the wide range 
of contexts, guides were individually tailored to each site* Each team was 
responsible for from four to seven sites, and each site was viaited up to 
five times during the two school years covered by the study* Information 
gathered during the site visits was used to develop a model (i*e*, a struc- 
tured summary) of the actual bilingual program at each site* Then, the 
19 aite models, plus information about tne USOS and TRC diffusion activi^ 
ties, were integrated into a complete model of the actual diffusion sys* 
tem* The intended diffusion system was inferred from USOS documents and 
from conversations with USOS personnel* 

2*1*4 What Were the Results of the Bilingual-PIP-Based Diffusion System? 

2*1*4*1 Adoption: What influenced potential adopter LEAs to adopt 
or not to adopt the bilin&ual^PlP projects? The nominal awareness activ* 
ities (mailing of literature* with follow-up phone calls by TRCs) produced 
very few adoptions * Instead, some LEAs responded to personal contacts 
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with stste or federal officialst Others vete seeking assistance in estab^ 
lishing bilingual prograias and contacted educators who were aware of the 
bilingual PXPSt 

Motivstions to adopt projects via PIPs included: 

• Interest in new ideas for existing programs* 

• Need for guidelines in establishing new programs* 

, • Belief that chances for Title VII funding vould be improved* 

Motivations not to adopt projects via PIPs included: 

• Lack of credibility of "project-replication" and/or "replication- 
via-packages" concepts* 

• Reluctance to participate in a national study concurrently with 
Implementing a new program* 

• Unwillingness to be restricted to operating at the grade levels 
specified in the PIPs* 

• Reluctance to accept federal funding because of strings attached 
and/or bureaucratic constraints* 

2*1*4*2 Adoption; To what extent were the projects chosen by 
adopters appropriate to these adopters? As sources of ideas and general 
guidelines, any of the projects could be considered appropriate to almost 
any district that had the required bllin&ual teachers * However* about 
half of the adopter sites did not have the bilingual teachers, and were 
forced to improvise without guidsnce from the PIPs* 

At a more specific level » such project features as instructionsl ob- 
jectives, materials, and team teaching were not slways appropriate to local 
conditions and were subsequently adapted (or ignored) to fit adopter^site 
needs and constraints* 

Among the Spanish-*?IP sites, the choice among the three PiPs was most 
often bssed on the proportion of English to Spsnish speakers as described 
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in the PIPst an issue of great importance to the site personnel* However, 
the instructional approaches described in the PIPs did not reflect these 
differing proportions, and to all intents and purposes^ sj^lection among 
the three Spanish PIPs was randott t 

2,1 • 4,3 Implefflentation) What factors fPIP. context* other) inf )"" 
enced imPleiLentation? Across all 19 field-test sites, the major e 
influencing implementation were existing local practices and condlw 
and state and federal regulations fetgt^ Title VIl) t Previous local ex- 
perience with bilingual education, project director and staff experience. 
TRC personnel, and locally hired consultanta were the other major factors 
in most si::est Several sites depended heavily cn neighboring LEA methods 
and materials to guide their project implementationt Some inexperienced 
sites leaned heavily on PIP management guidelines, and one site tried to 
follow the PIP to the better whenever possible* 

The lack of consistent USOE guidelines as to the nature and extent 
of pemissible adaptations led to considerable variation among adopter* 
site programs* Host &ites approximated the staff positions and organise* 
tion described in the PIPs* Virtually all combed the PiPs for good ideas 
that were suited to their programs* Overall^ however, we Audged the PIPs 
to be a minor influence on the field-teat site programa * This result was 
consistent with the results from Study 6. in which 12 compensatory*education 
PIPs were disseminated under roughly comparable conditions (see Section 
1*3*2)* 

2*1*4.4 Implementation! What vere the characteristics of the result* 
ing projects, and how closely did they resemble those described in the 
PIPs? The quality of the hilingual programs in most of the field^test 
sites was subjectively Judged by the RMC site visitors (most of whom were 
experienced bilingual educators) to range from sde^uate to excellent * 
Most employed sound, generally accepted bilingual teaching methods, al-^ 
though local conditions were more favorable to implementation in some 
sites than in others. 
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However, program quality was not the diffusion question ^ Rather, 
this study was concerned with replication of specific features* At a 
general level, any program consistent with Title VII guidelines (as were 
the f ield"test**^site programs) would resenofale those described in the PIPs* 
Beyond this general level, the resemblsnce wss minimal * However, the lack 
of replication should not be construed as a deficiency of the f ield^test- 
site programs* On balance, deviations from the PIP guidelines probably 
improved suitability to local conditions * 
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2t2 Substudv I: Conclusions snd Rec fwftfft'^ndatf^"" 

2t2tl A Systematic Approach to Diffusion 

The prlnclpsl contribution of the blllngusl-PIP field test va^ to 
reemphaslze the need (also demonstrated In the two compensatory-? IP field 
tests) for a a^tematlc ag&roach to _&lannlnflL diffusion efforta . The gen- 
eral finding from the three field tests was that the PIP-based Systems 
worked veil under some circumstances, but not under others. In the spe- 
cific esse of the blllngual-*PIP field test, all sites established (or ex- 
panded) bilingual programs and were generally satisfied with the results 
but, from a diffusion perspective, the PIP-based effort was not very sue* 
cessful. While the diffusion effort provided help to the sites In estab* 
llshlng their ovn programs. In no sense did the system cause exemplary 
projects to be transferred Intsct from one district to another. In shovt, 

the PIP-based effort provided technical ssslstance, but not diffusion of 

t 

projects. 

In order to understand why this vss so, and whst should be done to 
ensure more successful diffusion efforts, should they be attempted In 
the future. It Is helpful to think of a diffusion effort as a system 
made up of six major elements : 

It The diffusion goals set by the agency that plans the system* 

2, The projects/practices available for diffusion, 

3, The tar&et LEAs . with their specific attitudes, needs, and re* 
sources , 

4, *fhe delivery system for bringing the projects/practices snd 
the target LEAs together. Including personnel, materials 
(e,g,, PIPs), and all other resourcea, 

5, The Incentives for LEAs to (or not to) adopt projects and Imple- 
utent them accurately, 
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6, The competition faced by the system In the forms of laws and 
regulations^ other formal or Informal diffusion systems, and so 
on* 

The success or failure of a diffusion system involves a complex Interaction 
of all six eleskents: 

Dlf fuslon*System Goals 



Competing 
Ideas and 

. Gttldellneft 

, r , 

Ea:temal 
Incentives 

Figure 3* Simplified diffusions-system model. 

This simplified model of a diffusion system is used for discussion purposes 
throughout this report, A ukuch more elaborate model was used in the study 
for a systematic analysis of the diffusion effort (see Section 2,3, Sub* 
study I Products), 

In the three PIP diffusion attempts, these six elements were consi^- 
dered in isolation; some of them were not considered at all. The major 
recommendation from this study is that all six elements should be consl* 
dered explicitly and systematically , in planning or analyzing a diffusion 
study. The m & ,^or product of the study is a conceptual framework thst can 
facilitate such systematic planning or analysis, 

2,2,2 Detailed Conclusions ., and Recommendations Organized by the Six Ele - 
ments of a Diffusion Sys tep^ 

In the following six sections, study conclusions sre organized accord-* 
Ing to the six dlf £uslon<-s/atem elements listed above. Each section first 
summarizes the general conclusions of the three PIP field tests, then in- 
dicates the connluslons specific to this blllngual-PIP study* 
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2^2^2A Diffusions-system goals ^ At a general level, the tJSOE goal 
for all of the PIP diffusion efforts was to promote cost-effective, edu-^ 
catlonal Improvements for students, plus improved methods and procedures 
for school dlatrlct personnel* (Positive Impact on student achievement 
was a high priority outcome within the context of this general gosl, and 
Is discussed below under Subatudy II*) Goals specific to the FIP*dlf fusion 
efforts (as opposed to diffusion efforts In general) Included: 

A* Whole*pro1ect adoption : Diffusion of Intact , exemplary projects^ 
as validated by the JDRP* 

B* Rapid start-'UP i Intensive spring and summer preparation, lead- 
ing to fall Implementation at participating grade levels by the 
beginning of the school year* 

C. Stand-alone Packaging ; Minimal technical assistance* 

These goals wer& met to a certain extent In Study 2 (the prototype. 
compensatory-educatlon-PIF field teat) In which USOB provided strong 
monetary Incentlveii to follow the printed guidelines closely* However, 
these goals were not met In Studies 6 and 7 (the parallel, compensatory- 
and blllngual-educatlon-PIP field teats). In which adherence to PIP guide- 
lines waa less strongly enforced* Furthermore^ since monetary Incentives 
and enforced compliance with PIP guidelines are not consistent with cuf* 
rent USOE policy. It seema unlikely that similar diffusion systems will 
accomplish these goals In the future* 

In part» the failure to meet the gosls In Studies 6 and 7 were due 
to probJems moat appropriately categorized under the other five elements 
of the diffusion system. These problems are discussed In the following 
five sections* In Study 7, the bilingual PIP field test» additional 
problems were created by certain conflicts in the study gosls* These 
conflicts were of two types* The first type comprised the differences 
between USOE research goals and USOE service goals* Eesesrch goals re- 
quired strict adherence to PIP guidelines, while service to the .sites 
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often argued for extensive adapcacion. The result was a somewhat incon- 
siscenc USOE policy on adapcacion, wlch ^ consequencly large amounc of 
variance from sice Co sice in Che degree co which parcicular PIP-specified 
projecc feacures were implemenced. On balance, however, adapcacion was 
Che rule. 

The second cype of conflicc among goals involved differences becween 
USOE goals and TRC or LEA goals. The PlP^dif fusion goals were imposed 
on pre*'exiscing TRC goals by USOE on racher shore nocice and wlch liccle 
or no increase in TRC resources. The resulc was chac many TRCs were un* 
enchusiascic abouc che PIP diffusion effort. The LEA's goals included 
procuring funding and cechnical inpucs for cheir bilingual programs, and 
few, if any, of che field-Cesc sices subscribed Co Che three USOE goals 
lisced above. From che perspeccives of che school personnel, che RMC sice 
visicors, and even many USOE personnel, chese goals were of Cen seen as a 
hindrance Co escablishing che besc possible bilingual programs and^ in 
such cases, che success of che programs was usually given firsc prioricy. 

The ulcimace conclusion was chac che failure Co meec che chree PIP- 
diffusion goals lisced above was due, in pare, Co che face chac chese 
goals are noc appropriace excepc under very limiced conditions. In cho 
case of bilingual programs, chese goals vould, ac besc, be appropriace 
only for che diffusion of specific, highly scruccured and relacively sim- 
ple inscruccional or managemenc praccices. Such goals are noc appropriace 
for diffusion of complece, bilingual (or ocher) projeccs involving a com- 
plex of incerrelaced inscruccional, managemenc, and scaff developmenc 
componencs. 

Recommendacions on diffusion-sYscem goals . Based on che above con- 
clusions, we recommend Chat, in addicion Co che dif fusion-syscem changes 
recommended in che following five seccions, che chree PIP-dif fusion goals 
lisced above be revised as follows for fucure diffusion eflorcs: 

A. Uhole-projecc adopcion should be replaced by adopcion of separace 
praccices (i.e., componencs of projeccs) if clearly defined. 
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highly ettuctuted ptacticea eaciat* Otherwise, dlffualon of jrol - 
ecta ahould be abandoned in fflvor of dlffualon of general JPglncl" 
plea that can be applied to suit the needa of each district* 
With th3 poasible exception of somft specific instructional prac** 
ticea, the diffusion of general principlea ia probably the, moat 
relevant approach given the current status of bilingual education* 

B* Rapid startMip should be encouraged for practices that do not 

involve extenaive changes in staff attitudes or major development 
of staff skills* Bowever, in nkany LEAs and for many kinds of 
programs (including typical bilingual programs) long-term efforts 
to develop the readiness and capacity to implement new approaches 
are needed* For these programs and LCAs, rapid atart-^up is not 
feaaible* 

C* Stand"alone packaging (e*g*, a PIP) ia not effective except under 
very limited conditions* In general, a carefully planned balanca 
of packap^rig and technical assistance is appropriate* The details 
are determined by the characteristics of the educational practices 
and the target LCAs* This recommendation applies to all kinds of 
practices, including those in bilingual education* 

2*2*2*2 Projects and Practices * Perhaps the most important require- 
ment for a successful diffusion effort is to have useful, effective prac- 
tices to disseminate* In the opinion of the authors, the process of 
selecting projects or prscticea for diffusicxi efforts has been grossly 
insdequate, not only in the PIP diffusion efforts but in most other diffu- 
sion efforts with which we sre fomllisr* This problem hss been reflected 
In the fsct that the recent diffusion literature and diffusion conference a 
have focused almost exclusively on dissemination strategies and evaluation 
of results, with little or no concern shown for the needs of the LEAs or 
for the characteristics of the projects snd prscticns avsilsble for dissem* 
ination* 

The projects chosen for PIP-psckaging were selected from among exem- 
plsry programs opersting in school districts* Selection techniques did 



mit aep&rate context effects from project effects nor were project char* 
acteristics analyzed adequately in terms of target LEA needs and resources* 

Many truly exemplary programs can provide little in the way of pro* 
cedures or materials that (a) can be readily implemented in other school 
diatricta, and (b) will produce major improvementay Projecta and prac* 
tices vary widely in their auitability for diffusion* 

• Project factora favorable to auccessful diffuaion: 

— Inatructional rather than management focua in the project* 

— Self "contained! encapaulated projecta affecting only a few 
atatf members* 

*^ Specific, structured procedurea or materials* 

— Relatively atable approach (not rapidly evolving)* 

*^ General availability of required peraonnel akilla and other 
resources* 
New ideas* 

• Project factors unfavorable to successful diffuaion: 

~ Management rather than instructional focua in the project* 

— Whole^achool, whole^day projects* 

— General, flexible guidelines for operation* 
Dynamic, evolving practicea* 

Requirementa for exceptionally skilled staff or elaborate 
materials, equipment, or facilities* 
Conventional ideaa applied unusually well* 

The bilingual projecta choaen for diffusion presented major problems 
from a diffuaion perspective* These projects were chosen more for their 
apparent achievement impacts than for their features with the result that, 
while the original programs were probably all very effective and may have 
shown conaiderable variation in their original contexts, there were very 
few real differences among the packaged projects * In general, the follow- 
ing comments apply to all: 
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• All four projects depended heavily on good blllggual teache rst 
This drastically reduced the number of potential adoptera for 
which the projects were suitable, since an adequate supply of 
good bilingual teachers Is not readily available In many LEAs 
(although this feature did not keep sites that lacked such 
teachers from applying for PIPa)« 

• Thfe projects were relatively minor varlatlona on atandard themes * 
Thus, while experienced sites &ay have found some good Ideas, 
they should not have expected any major Innovations* Sites with 
no bilingual-education experience could appropriately have ex- 
pected baalcally sound advice, although comparable advice waa 
available from a multitude of other sources « 

• Much of what distinguished the orlglnlal, exemplary projecta 
involved Project management * While this Is a major factor dls* 
tlngulshing good programs from poor ones, many management fea-- 
turea are not readily transportable, since they require changes 
to flnnly established LEA organizational and administrative 
structures* 

• Instructional features of the projects were generally flexible 
In the developer sltea« The general guidelines that were avail* 
able were of only limited help to either experienced or Inexper- 
ienced adopter sites* 

• The projecta, as packaged, were several year6__pld by the time 
they reached adopters* Materials, and In some cases methods, 
were often viewed by adopters as out of date* 

In short, the LEAs needed specific, current ideas for methoda and 
mat^rlal^ that could be used In the clasaroom hv the a vailable teachers , 
and the four projects, as packaged, did not provide much help in these 
areas* The success of the projects In the developer sites appears to have 
been due largely to the exemplary application of sound management and 
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instructional principles by talented and dedicated staff in a supportive 
environment t These are primarily context features and thus are not read** 
ily diffusable* 

Recommendations on choosing Projects and Practices t Projects or 
practices must be carefully analyzedi^ and selected to match the needs and 
resources of the target LEAS t For example, while the four bilingual proj- 
ects chosen for PIP packaging depended heavily on good bilingual teachers 
following general guidelines, many field*test sites needed specific in- 
structional procedures that could be used by monolingual English-speaking 
teachers with Spanish /Frenches peaking aidest 

Adoption of whole, intact projects is reasonable only for encapsu- 
lated, instructional projects Ce,g,, a pull-out, reading lab). Projects 
affecting the operation of the school as a whole shou ld be treated as a 
collection of components (management components, instructional components, 
parent involvement components, and so on) that can be disseminated as 
separate units. Typically, an exemplary bilingual program is made up of 
many such highly interdependent and con text "determined components and 
cannot be diffused as an intact unit, 

2,2,2,3 target LEAs , Logically, the place to begin in designing 
any diffusion system is with the educational needs of the target-L£A stu- 
dents, and with the objectivea, available resources* and other relevant 
characteristics of the target LEAs, A major * additional consideration 
in a diffusion system is the size of the target audience for a particular 
educational project or practice. These LEA-relaced factors are discussed 
below. 

Student needs . Student characteristics (and consequently student 
needs) varied widely from site to site and also within each site, Lan^ 
guage skills ranged from extremely limited proficiency up to native pro** 
ficiency in both languages. Some students were clearly dominant in one 
language, a few were highly proficient in both languages, and still others 
were lacking adequate proficiency for academic purposes in either language, 
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Scores on standardized reading and math tests ranged from well bslow to 
well above averagst In general, the PIPs did not provide adequate guide* 
lines for dealing with this vide range of students* 

LEA adoption oblsctives t It is not safe to assume that all LEAs are 
interested in making changes » and even those LEAs actively seeking change 
may not want to install completely ugv programs* Many of the LEAs involved 
in the three PIP field tests were more interested in refining existing in- 
structional practices, or providing more effective management structures 
for loosely organized programs* Many were also motivated, in part, by the 
federal funds or technical assistance associated with the projects being 
disseminated* For these LEAs, it is important that the available projects 
or :&ractices fit in vith existing inst r uctional and management approaches t 
Even those LEAs that wished to add completely ne^* urograms (or replace 
existing ones) vere seldom willing to consider major changes to district-* 
wide practices, or policies* 

LEA resources * Resources required for the adoption of new projects 
may include personnel, materials, and facilities* Staff skills and atti* 
tudes are perhaps the most important LEA resources required by the bilin* 
gual PIP projects, and at least half of the sites could not obtain a com* 
plete staff of teachers with the skills and attitudes specified in the 
PIPs* Thus, these sites could not accurately implement either the PIP 
projects* or any other projects requiring a full staff of skilled, billn^ 
gual teachers* Instead, these sites needed projects designed for the 
personnel that were available* In the ehort run, this usually meant mono* 
lingual, English-speaking teachers with bilingual aides* In many LEAs, 
a long-term program of staff development was indicated. 

Other relevant LEA characteristics * Organizational and administra- 
tive structures in the LEAs were also key features from a diffusion stand- 
point, and they proved highly resistant to change * Most administrative 
procedures* including staff hiring policies and management of federal 
projects, were applied uniformly within a given LEA, and exceptions were 
not normally made for single projects* This precluded implementation of 
many PlP-project features related to these management procedures* 
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In addition^ the attitudes of key administrators are critical 
diffusion-system considerations* Attitudes toward bilingual education 
and whole-project adoption varied widely among the bilingual -PIP-fleld- 
test sites* In at least some sites, there were no enthusiastic PIP- 
project aupporters In the administrations* and the PIP project directors 
had to fight an uphill battle. 

The overall Implication Is that new projects must fit the attitudes 
and exlstltig organizational structures unless long-term educational and 
capacity-building components are Included In the diffusion system* A 
PIP-type diffusion system does not appear to be a feasible mechanism for 
changing these structures and attitudes. 

The community contexts can be Important factors, but In the PIP field 
tests they have been generally appropriate for the PIP projects. Often^ 
principals In the bilingual PIP sites reported active Interest from the 
community, and some told of parents calling to find oul how to get their 
children Into the programs. 

A final key LEA characteristic^ from a diffusion perspective, Is the 
extent to which LEAs seek out Information . Among the LEAs with needs for 
bilingual programs^ many of those that actively seek out Information al- 
ready have programs In place. LEAs that have only rudimentary bilingual 
programs or none at all tend to be those that are not as active In seeking 
Information about new programs* Thus, a much more aggressive form of 
awareness activity Is needed to reach this Important part of the target 
audience than was provided In the bilingual -PIP field test. 

Size of the tarRet audience . An additional diffusion problem con- 
cerns the size of the target audience, that Is, the number of target LEAs 
for a given project or practice* In the bilingual PIP field test, the 
audience was broken Into two groups—those LEAs with French^-s peaking stu- 
dents^ and those with Spanish-speaking students. While the number of 
Spanish-speaking students In the U.S. Is large. It appears that the number 
of LEAs with students from French-speaking backgrounds Is relatively small. 
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Thus, while elaborate, PlP^type materials might be Justified for Spanish/ 
English projects, the PiPs could not be considered cost effective in the 
smaller market for French/English projects* A tsrget sudience of this 
size would suggest s quite different diffusion approach, probsbly involving 
less elaborate inaterials and a correspondingly it^creased level of technical 
assistance* 

Recommendationa on analvzina target LRAs * Both the characteristics 
and the size of the target sudience should be clearly understood before 
plsnning a diffusion system: 

# The LEAs with the greatest needs msy require swareness snd 
capacity^building efforts rather than project^dif fusion efforts* 

# The size of the target audience should be taken into account in 
plsnning diffusion systems* In general, elaborste materials snd 
technical assistance strstegies will not be cost-effective for 
small groups of target LEAs* 

2*2*2*4 Delivery system * A delivery system includes all personnel, 
msterisls, and other resources used to bring the projects/prsctices and 
the tsrget LEAs together* The delivery system must provide awareness 
plus technical support for selection gnd implementation of the Projects/ 
practices * In this study, the major components of the delivery system 
were the piPs and (secondarily) the TRCs* 

The three pip fiela tests have led to the following general conclu- 
sions concerning delivery systems: 

# An effective delivery system cannot be designed in isolation 
from the total diffusion system* That is, the deliver/ system 
cannot ignore the characteristics of the diffusion-system goals, 
the projects, the target LEAs, the incentives, snd the competi" 
tion* 
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• Printed materials i^*i*t PIPs) are not read carefully In the LEAs 
unless atrong Incentives are provided* In general, printed 
materials (whether In PIP or other formats) are veil suited to 
providing reference Information, but not to: 

* focufttnfl attention on key project features 
persuading adopters to try new Ide^s 

- Identifying problems and providing feedback 

- providing reassurance 

all of which are essential dellvery-^system functions* Other 
media (e*8*t film, videotape) appear to have some of the same 
problem!}, although they were not evaluated In this study* 

e While some level of packaging Is probably an Important component 
or moGt delivery systems, elaborate packaging (e*g*, PIPs) Is 
expensive and time consuming* It Is cost effective only for 
large pools of target LEAs, and for projects/practices with long 
shelf lives* 

• People (l*e*, disseminators) are much more effective than mate^ 
rials In dlrectir.g th2 attention of the target LEAs to essential 
points and providing feedback when things go wrong* 

• PIP*-based diffusion systems would be most effective for wide* 
spread diffusion of structured projects or practices accompanied 
by large monetary Incentives to replicate* However, the use of 
monetary Incentives In this manner Is not consistent with current 
federal policy* 

These generalizations apply equally to delivery systems for bilingual 
or non-blllngual projects/prsctlces* A need for complementary efforts 
In local capacity building , while not unique to bilingual education, should 
be a special concern here due to the short supply of experienced, bilingual 
personnel In many target LEAs* 
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The blllnftual'PIP delivery aystem t From a diffusion perspective, 
the delivery system for the blllngual-PIP field test was not very effec- 
tive* Given the four PIP projects and the target LEA characteristics, It 
seems unlllcelv that any dellveOL_aystem could have resulted in large num * 
bers of faithful replications . However, If an appropriate match of proj- 
ects and target LEAs had existed, the following delivery-system problems 
would have limited the effectiveness of the diffusion system* 

The PIP materials . Including the awarenesa booklets as well go the 
PIP manuals , were not particularly well matched to the projects, TRC dls-* 
semination activities, or to the Incentives provided by the system* Un- 
like the cotrpensatory-educatlon projects, which were mostly straight- 
forward, single-subject. Instructional practices, the blllngual'-educ&tlon 
projects uere day-long, multiple-subject. Instructional, management, and 
staff development systems that required substantial accommodation and 
organizational change from adopters* In practice, the Implicit assumption 
that the PIP concept was applicable to bilingual education was aimply not 
well founded* 

Instead of the stand-alone PIP formet, materials might better have 
been designed for use In TRC-run workshops, or as reference materials to 
be used after completing such workshops* Under thiise conditions, the ma- 
terials could have been simplified and shortened aomewhat as compared to 
the PIP manual£t, leaving the TRC presenter to tailor the workshops to the 
specific needs of the audience* 

The workshop format would have also Increased the likelihood that LEA 
personnel would study the content of the msterlala carefully, something 
that did not occur with the PlPa* As noted below, the delivery syatem 
provided no real Incentives to read the materials thoroughly, and much of 
the existing PIP materia)! >^'zs wi dely unused* Project directors reported 
that they liked the Idea having* jill project guldellnea conveniently 
packaged, but few project directors or teachers referred to the PIPs 
extensively after the projects got underway* 
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The dlffuelon personnel , primarily disseminators from the TRCs, had 
three Important functions In the field tryout: (a) promotion of the 
concept of adopting an exemplary project; (b) assisting In selection of 
an appropriate project, and (c) provision of staff training to LEAs* In 
all three Inunctions, they were severely hampered by lack of training and 
resources* Hany of the TRCs actively supported the PIP field test within 
the llmxts of their resources* Some, however, were unenthuslastlc about 
the PIP diffusion approach and did little more than was required of them* 
In general, the latter group produced no adoptions* Clearly, successful 
disseminators must have adequate funds and trslnlng as well as positive 
attitudes toward their taska* 

The blllngual-PIP delivery system Ignor other existing laformatlon 
and resoyr^*^ networks that could have been U' ' .ized to disseminate the 
PIPst The use of these listing, informal and naturally^occurring net^ 
works might have produced PIP adoptions from a different set of LEAs or, 
minimally, could have bestowed additional credibility and redundancy to 
the TRC dissemination effort* 

The results in the bilingual PIP study were that the nominal aware- 
ness activities produced verv few applications * Selection of projects was 
essentially rand o Some TRCs provided excellent staff training , but it 
was not alwavii „ cvant to the PIp guidelines , 

Recommendations on developing delivery systems t A delivery system 
must be designed sys t ema t Icallv within the context of the complete diffu- 
sion system (goals, practlcea, target LEAs, delivery system, incentives, 
competition)* Section 5*2 of this volume, together with Volume II of this 
report, describe a conceptual framework Intended to facilitate systematic, 
diffusion-system deslgnt 

In general, a delivery system will require both people and packaging 
(although usually not PlP^type packaging)* However, the functions of the 
people and th:: materials (and therefore the nature of the materials) will 
depend on the other parts o. the diffusion system, with some systems 
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requiring relatively more emphasis on technical assistance and others 
requiring relatively more emphasis on materials* 

2t2t2t5 Incentives t The PIP studies and other diffusion studies 
have shown that LEAs must be strongly motivated before they will make 
changes* The desires of LEA personnel to improve pro^fraias, and legal 
requirements to provide new services, obviously lead to some kinds of 
changes* However, these kinds of incentives uaually do not motivate 
adoptera to read printed materials carefully or to follow guidelinea 
closely* 

Certain external incentives*-- specif ically the USOE requirement of 
detailed compliance with guidelines as a condition of funding~proved 
effective in the original, compensatory*education*PIP field test (Study 
2), but continuing use of funds in this msnner is not consistent with 
current USOK policy* 

TtprnmmAndations on considering incentives * A reasonable expectation 
thst target LEAs will be motivated to adopt the projects or practices 
should be a prerequisite for developing a diffusion effort* Assuming that 
strong, external incentives to replicate accurately will not be provided 
(and probably would not be appropriate), substantial adaptation of proj- 
ects/practices (bilingual or otherwise) should be anticipated* 

2*2*2*6 Competition * A diffusion system intended to promote adop- 
tion of specific projects or practices cay be in direct competition with: 
(a) federal regulations or laws; (b) state regulstions or laws; (c) local 
regulations, policies, and existing programs; (d) other LEAs promoting 
alternative practices; (e) professional organizations promoting alterna^ 
tive practical; (f) commercial organizationa promoting alternative prac^ 
tices; and (g) other diffusion systems promoting alternative practices* 
All of these sources of competition affected the bilingual*PIP field test, 
and reduced the impact of the PIPs* 
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HArnmnpT xdatlons on atvalyzing_the comPetltlon t The alternative proj" 
ects/practlces and their sources must be understood If the diffusion of 
specific projects or practices Is to be successfult For each potentially 
competing sourcet a decision Is required as to whether to adapt to the 
confllctitts position (often required where laws are Involved) or to com- 
pete directly* This principle holds In all areas of education, but bllln** 
gual education Is affected by an especially wide variety of laws and regu** 
latlons that must be taken Into account In planning a diffusion effort* 

2t2t3 S^jfT^rtrv of the Dlf fuslon-SubstudV Conclufllons and Recommendations 

242t3tl Conclusions t The basic conclusion from the process evalua** 
tlon was that while good bilingual programs were established In most of 
the field-test sites » and the blllngual-?I? diffusion effort helped the 
sites to establish these programs, the PIP diffusion effort wss not veicy 
effective In gei^tilng substantial numbers of target LBAft to implement the 
projects described In the PIPs* Relatively few LEAs applied and» In the 
19 sites that received PIPs snd Title VII grants, program features were 
determined more by locsl factors and Title '/II regulations than by the 
PIPst Most of the LEAs felt that extensive adaptation of the projects 
was required and. In general, RHC concurred* 

The basic system breakdowns had already occurred before the fle^d" 
test: sites became aware of the PIPs . The major breakdowns occurred In; 

• Establishing system goals* 

• Project selection* 

• Project packaging* 

• Diffusion agent selection and training* 

2*2*3*2 Recommendations * A systeiaatlc approach to planning dlffu" 
slon efforts Is needed, and should Include: 

• Establishing specific, practicable goals * 

• Selecting and validating appropriate practices * 
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# Analyzing target LEA needs* 

# Developing a delivery ayatem , with an appropriate balance of 
coordinated technical aaalatance and packaging* 

# Providing Incentlvea , If needed* 

# Conalderlng the competition , and adapting the dlffualon ayatem 
aa neceasary* 

The PIP concepta (Section 1*1*1*2) are not aultable for the diffusion 
of complete^ blllnfjual prolecta * Specifically: 

The PIP Concepta : Should be replaced by : 

Stand-alone packaging Complementary packaging 



The PIP concept may be appropriate for the dlffualon of aome atructured, 
relatively aelf^^'contalned componenta of bilingual (or other) projecta* 

Finally, detailed dlf fualon-^ayatem recommendations depend on the 
goala that are aet for the ayatem* In particular, a ayatem Intended to 
promote the dlffualon of aPeclflci validated Projecta or practices with 
little or no adaptation will be very difficult to eatabllah, and In auch 
a ayatem, each of the alx elementa llated above repreaenta a major prob*- 
lem area* By contraat, a ayatem that will simply provide target LCAa 
with acceaa to new Ideaa and to technical aaalatance and which Permits ex* 
tenalve adaptation la relatively eaay to eatabllah* 

Given the typea of projecta and practlcea available fo. ',ffualon 
(bilingual and other), the former type of system appears to be unwarranted, 
and the latter Is recommended as a more practicable alternative* However, 
we do not suggest that the two alternative typea of systema are In any uay 
equivalent* In the latte'' type of diffusion syatem (i*e., one that permits 
extensive adaptation), results In the target LEAs will vary widely, and 
expectations for the Impact of such a system should be tempered accordingly* 
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2^3 Products of Substudy I 
2,3^1 A Fr^T»^" ork for Planning or Analyzing a Diffusion System 



A major product of the diffusion substudy Is an approach to develop- 
ing frameworks or '^models** of diffusion systems^ The approach Is specify 
Ically Intended for syst&ns designed and operated by agencies such as USOE 
or SEAd for large scale diffusion of projects or practices. The frame- 
works are Intended to: 

• Integrate the goals^ projects^ target LEAs* delivery system^ 
Incentives, and effects of competltlon t 

• Include the processes of Information transfer* decision making, 
and resource allocation t 

• Focus on the actions available to the system operators (etg*, 
USOE) to achieve systra goals, such as selecting and training 
diffusion personnel, developing guidelines and materials) and 
allocating resources t 

This approach was Initially develope^^ by KMC for Studies 2 ^nd 6» and was 
extensively revised In the bilingual PIP studyt It Is described In more 
detail In Section 3*2 of Volume 11* Appendix E In Volume II comprises the 
complete model of the Intended system as used In this study* 

2*3*2 A Comparative Analysis of the Four Blllngual-PIP Projects 

This product Is Intended for the educator who wishes to know. In 
some detail, what practices are described In the four bilingual PIPs* 
Since fhe PIPs are designed as step-by-step, how^to-do^lt materials. It 
is a laborious process for the reader to extract these practices and even 
more difficult to compare one PIP with another* this comparative analy- 
sis should provide an efficient guide to the actual PIP contents* 

The comparative analysis of the PIPs Is Included as Appendix C In 
Volume II* 
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3t SUMMARY OF SUBSTUDY II: IMPACT ON STUDENTS 



3tl Introduction to the Impact SubstudY 

3tl,l Rationale for SubstudY II 

The ultimate goal of all of the PIP*dxffu6lcn efforts tias been Im- 
pact on students, especially Improved achlevementt Thus, the Investlga* 
tlon of Impacts on achievement was the major focus of this substudyt It 
should be kept In mind, however, that the Primary focus of the entire t 
two*vear. blllnftual^PIP study was on diffusion . Achievement impacts were 
treated somewhat more extensively In the original field test of six proto* 
type compensatory*educatlon PIPs (see Study 2, Siictlon It3t2)t Achieve^* 
ment was not examined at all In the later field test of 12 compensatory* 
education PIPs (Study 6) that ran In parallel with the blllngual*PIP field 
test* 

The Impact substudy received secondary emphasis In the blllngual-PIP 
study because: 

• It was not certain In advance that the diffusion system would 
produce the complete Implementation of the projects In the target 
tEAs (and. In fact. It did not)* Thus, It was not certain that 

a major effort should be Invested In student outcome evaluation* 

• Because the study paralleled only the first two yesrs of a five* 
year project Implementation plan, data could only be collected 
from grades K-2 of projects that muld eventually expand to fourth 
grade* Thus the study could not provide student outcome results 
from completely Implemented, fully operational programs* 

On the other hand. If USOE had waited two years to obtain the diffu- 
sion results before considering an outcome evaluation, much of the data 
needed for a longitudinal Impact assessment might never have been collected* 
Consequently, It was reasonable for USOE to request a limited form of 
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outcome evaluatioii» and this outcome evaluation was the basis of Sub&tudy 
II* Before describing this &ub8tudy» however^ it is necessary to conei-* 
der tvo technical* evaluation distinctions that are central to the prob* 
lems encountered in the substudy* 

3*1*2 "Impact^ versus "Performance Level" 

This report refers repeatedly to the impact on achievement of a dif- 
fusion system* a program* or a project* By this va mean the change (hope- 
fully Improvement) in performancs produced by new practices* The impact 
question is emp( lized here because this was a diffusion study* and the 
Justification for spending time and money on diffusion derives from the 
change that resttilts* Impact on achievement ia not the only impact of ic- 
threat* but it is a special concern of many educators snd diffusion per- 
sonnel* and it is emphasized in this report. 

Impact on achievement may also be of interest to the LEA* because 
mo^t LBA decision makers would like to know whether changes in practices 
are followed by improved performance* However^ the more important achieve* 
ment consideration from the LEA point of view may be the performance levsl 
of the students* "Performance level" refers here to whether studenta are 
achieving well or poorly in relation to other groups of students or in 
terms of requirements for jobs and personal oatisfaction * An LEA may not 
be completely satisfied with a large impact if the performance level is 
still low. Conversely* the LEA may be entirely satisfied with a small* 
positive impact if performance levela were already high* 

From the standpoint of the dif fuslon^^system planner * however* Impact 
is the primary consideration * High performance levels are no Indication 
of diffusion*systCT success* since they do not necessarily prove that there 
has been improvement due to the diffusion effort* Conversely* low perfor- 
mance levels do not necessarily imply diffusion*^system failure* since they 
may represent Improvement over still lower levels of performance. The 
distinction between student performance level and dif fusion-^system impact 
is illustrated in Figure 4. 
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Figure 4« Performance level versus impact* 



Thus, to the extent that student achievement ou'rcomes are addressed 
in this diffusion study, tha emphasis is on impact * Performance level Is 
also discussed, but has only indirect relevance to the study* 

3*1*3 "Apparent'^ Versus ^'Actual" Impacts of Exemplary Programs 



In terms of the above diagram, one might naively sssume that positive 
impacts would result from the successful diffusion ol exemplary projects 
(i*e*, in the PIP studies, projects developed and implemented with appar-^ 
ently positive impacts in the originating sites)* Of course, it has al* 
ready been noted (Section 1*1*1*3) that a single project in two different 
contexts leads to two different programs * However, an additional factor 
"'the evaluation (and, in particular, the variation or error in evaluation 
procedures )»is involved in the apparent impact of a project on achievement* 
This distinction Is illustrated in Figure 5* 
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Figure 5* Factors affecting apparent impactt 



The problem related to the evaloatlon component of the apparent 
Impact la that measurement of program Impact in the re«l world l0 very 
Inexact* Because of the large amount of error variance* two different 
evaluations of the Identical program may produce very different results* 
Further* the amount of error In a typical evaluation Is large In compar* 
Ison to typical program Impacts on student scores (up to several times 
as large* Horst* 1978)* Finally* the separation of project Impacts from 
cont^t Impacts Is next to Impossible In many LEAs* 

The net result 18 that (a) even the moat faithful replication of an 
exemplary project in a new site Is not likely to reproduce the original 
pro g ram * and (b) even If the programs were Identical* differences (errors) 
In the evaluations of the originating and replicating sites would make It 
unlikely that the apparent Impact of the project vould be the same in the 
o rl Rlnstlng and rePllcstlng ^sltee * The sctual Impact of a project la an 
elusive concept and* In practice* Is very difficult to determine under 
field-test conditions* 
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3*2 Scope and Results of Substudy II 



3*2*1 What Were the Impact Substudy Questions? 

The impact evaluation focused on improvements in student achievement 
resulting from the diffusion effort * Affective impacts were also addressed, 
but to only a limited extent due to the technical problems involved* The 
major subject areas addressed were: 

• English reading 

• Spanish/French reading 

• English oral language proficiency 

• Math 

The impact evaluation did not focus on student Performance level 
because, as explained abcve, performance-levei measures are not directly 
relevant to diffusion questions** Furthermore, the performance level of 
the students does not indicate whether the bilingual programs are more 
effective than alternative approaches , unless the alternatives are com*^ 
pared with rhe bilingual programs utider rigorously controlled cotiditions* 
Such conditions were not available in the field-test sites* 

As the Scudy progressed and the difficulties iri answering the impact 
question in a bilir-gual-^project field test became increasingly apparent, 
RMC's ittpact-subrtudy emPhaais shifted toward methodological issues * The 
By the end of the study, major impact questions had become: 

• In general, how, if at all, is it possible to determine the 
impact of new educational programs in real-school settings? 

• Specifically, are there credible impact results (either positive 
or negative) from any of the field-test sites? 



^However, performance-level data, which were provided by the field- 
test sites, are reported in Appendix D for the Interested reader* 
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3*2*2 What Was the RMC Role in the Impact Substudv? 



RHC had two* distinct roles In the impact substudy: 

• Consulting i«ith locsl evaluators and project directors in order 
to make the evaluations as technically sound as possible. 

• Interpreting and Integrating results from the local evaluations* 

RHC was not expected to collect data or to perform the original 
snalyses on the dats» and all decisions were ultimately made by the in* 
dividual sites* 

3*2.3 What Was the Research^^te^roach to Anawering the Impact Questions? 

The basic intention in the impact substudy was to develop the beat 
possible quasi-*experimental designs in each LEA and» in particular* to 
obtain baseline and comparison*group data« ^erever possible^ that could 
be used to establish impacts* 

3*2*3*1 Ma1or Problem areaa * The first year of the study was in- 
tended as a development yedr for impact substudy* As the year progressed* 
it became clear that there were seven major problem areaa that would in- 
validate the evaluations for the purpose of 'determining impact * These 
problra areas were: 

• Comparison standards : Xn most sites* true control groups were 
Sxmply un^t^ailable due to state and federal lauSt and few LSAs 
maintain historical comparison data that are suitable for estab- 
lishing program impact. Various comparison groups of dissimilar 
students in dissimilar programs were available for performance- 
level comparisons, buu 41fferences between such groups and the 
bilingual-progr^^ students did not necessarily imply impacta due 
to the ney bilingual Progjrams * Many of the achievement tests 
used by the sites provided norms and thus made nom*ref erenced 
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evaluations possible. However, KMC had strong doubts about the 
relevance of available test norms to the program students and, 
in any case, recent evidence suggests that norm^based comparisons 
are very imprecise for any single school district. 

• Technical adequacy of tests . Most of the sites used standardized 
tests for English reading and math* These tests can usually be 
considered technically adequate for their intended purposes , but 
may not br suitable measures for bilingual programs. 

In addition, the correct level of the test must be selected* 
Levels that are too easy or too difficult provide inaccurate 
measures* Although appropriate levels were not systematically 
established in most sites, floor and ceiling effects were seldom 
encountered among second grade students in the second year of 
the study* 

French/Spanish reading tests, language proficiency tests, and 
affective tests typically exhibit serious technical problems , 
such as lack of adequate reliability. Measurement in thase areas 
was a major concern for all sites* 

• Description of student characteristics : Many sites did not 
provide adequate descriptions of student language skills and 
backgrounds, but it was clear that widely differing students 
were grouped together in most analyses* For some sites, BHC 
could not interpret results because of insufficient information 
as to whether the results applied to students who were limited 
English speakers or fluent Engliah speakers* 

• Curriculum versus test*content match . RMC needed to know, at 
least, whether there was a match between the tests and the cur^ 
ricula at the general subject level (e*g*. Were students who took 
English***reading tests actually studying English reading at all?) 
Even this basic information was not available for individual 
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students within a site» although RHC collected ouch information 
at the classroom level* Ideally, gome indication vaa needed that 
the test covered the specific areas emphasized in th* currlculum t 
This information was not available from most of the sites* 

• Testing_and scoring Procedures * Appropriate procedures for 
testing and scoring present no real technical problems but, in 
practice, insdequate procedures are common sources of evaluation 
errors * Many sites experienced difficulties .in these areas* 



• Data analysis ; There are many unresolved issues in data analy- 
sis. However, the major concern was f^hat many sites were not 
applying basic, generally accepted analysis procedures, includ-* 
ing: 

Use of appropriate analysis models* 
~ Use of appropriate test scales (e*g*, standard scores)* 
— Grouping students by langusge skills* 

Including only students with both pre-* and posttest scores^ 

and analyzing dropouts separately* 



• Interpretation of result s: The final question that must be 
answered in an evaluation is: Given an apparent impsct, is it 
reasonable to believe that it was due to the Program? 



Answering this question requires, at a minimum, a rudimentary- 
description of the instruction received by each student, and 
few of the sites provided this information* While some addi- 
tional information was collected by RMC on instructional treat- 
ment at the classroom level, it was adequate for only the most 
general level of interpretation* Only three of the second-year 
reports presented any attempt at in-depth interpretation of 
results* 



3*2*3*2 RMC imPact-evaluation activities * Over the course of the 
irst year, RMC organized two workshops for project directors and evalu- 
tors, and consulted with each site via telephone and during site visits* 
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Then, prior to the second y^ar, KMC worked with the local evaluators^ most 
of whom were highly skilled and experienced, to develop specific recommen" 
dations for the second-year evaluations for each site in each of the seven 
problem areast Vhere they were implemented, these secund-year designs 
eliminated many of the first*year problems, but the first two problem 
areas, the unavailability of adequate tests and the lack of appropriate 
comparison groups, were beyond the control of either LEAs or RMC* 

Because of the problems encountered in the above seven areas, the re* 
search approach dhif l«^d to focus on the prior question of the credibility 
of the impact results . A systematic (although basically subjective) pro- 
cedure was developed for evaluating the credibility of impact findings from 
the field-test sitest This procedure consisted of rating each evaluation 
(each subject area in each site) as to how well the impact-evaluation prob- 
lems had been solved in each of the seven problem areas t 

All lo'^ai site evaluation reports were examined and, in addition, 
BMC obtained raw data and conducted extensive addiLtional analyses for 
selected siteSt Second year, second-grade data were examined first be- 
cause second-grade data normally suffer ii^ss from the problems of testing 
young children than d( kindergarten and i:Lrst<*grade data, and thus could 
be assumed to provide an upper limit on the precision of the available 
results* 

The evaluations of ^11 four subject arenas (English reading, Spanish/ 
French reading, oral English language proficiency, and math) were rated for 
each site as to the extent of the inq) act-evaluation problems in each of the 
seven areas: 

• Comparison standards* 

• Technical adequacy of tests* 

• Description of studeut characteristics* 
1* Curriculum versus test-content match. 

<f Testing and scc/ ing procedures^ 
^^ta analysis • 

• cerpretation of results. 
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The overall credibility oC the impact results vas held to be no more 
credible than the weafce9t link in the seven areas* 

3*2*4 What Were the Results of the Impact Substudv? 

The results were that only one of the 19 evaluations was wen some* 
what convincing in terms of impact * The remaining evaluations simply 
provided little or no information as to whether or not there had been an 
impact (although two of the other evaXuatoars provided excellent analyses 
of the problems encountered, and on.'^ of these developed an exemplary 
longitudinal design that okay answttr the impact question in the future)* 

None of the sites was given a "fully convincing** rating in any of 
the seven areas, but the worst problem area was that of comparison 
standards * In most L£As» it appears that there is no way that a bilingual'- 
program evaluator can obtain the necessary background and comparison data 
unless these data already exist as a result of an ongoing, district-wide 
evaluation program* Even where district-wide programs exist* there is no 
guarantee that the available information will be suitable for answering 
the impact question* 

In general » English reading and math received the highest impact- 
credibility ratings ^ The additional problans of measuring English lan- 
guage proficiency and French /Spanish reading msde the credibilty oC 
Impact results even lower in these subject areas* 

Evaluations of impact on student affect and ps rent /community in- 
volyement were also examined, although in less detail* Student affective 
impacts are» in general, ev^n more difficult to assess than are achieve- 
ment impacts, and no definitive conclusions were reached in this area* 

The success of the parent/community involvement components are gen- 
erally easier to determine, at least at th& level of counting attendance 
at meetings and ; .^^ng participation in the classrooms* Most sites 
could report ndvsnces in these areas* 
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3,2,5 Problemg with LEA Evaluation Reportg 



While dome of the problems with the eve luat lone were beyond the 
control of the local evalvitors, the problems with the f ield'-test-^site 
evaluation reporte , however, are matters of some concerut Evaluation 
reports serve a variety of purposes (etgt, compliance with federal regu- 
lations, auditing, reporting to LEA decision makers and school boards) 
and may influence the development of recommendations for program changes* 
Many of the reports provided documentation of staff development, parent 
involvement, and some description of program implementation, and thus, 
would be useful for some of these purposes* However, with three notable 
exceptions, these reports, like most LEA evaluation reports, were incom- 
plete and/or misleading to the extent that they were inadequate for any 
achievement-outcome evaluation purpose, performance*level assessment as 
well as impact measurement * 

One of the exceptional rep^ ts described a well-reasoned design and 
analysis that did provide a great deal of locally useful information as 
well as some evidence on achievement impacts* A second reflected an exem- 
plary, longitudinal design that should provide both impact evidence and 
other locally useful information in future years, slthough data were not 
available in time for this report* The third provided an excellent analy- 
sis and diacussion of the evaluation issues and problems, but problems 
beyond the control of the evaluator precluded any impact evidence snd 
severely limited the value of the evaluation for any local purposes* Two 
additional reports shoved noteworthy attempts to establish appropriate 
designs that may well produce valuable information in future years* 

Problems aiiK)ng the temaining reports included incomplete information 
about—Ca) student characteristics, (b) numbers of students, and (c) tests 
and levels in use* Many reports also used inappropriate scales (e*g , 
grade-equivalent scores) and inappropriate evaluation models (e*g*, post- 
test minus pretest) (Horst, Tallmadge, and Wood, 1975)* In fact, some 
reports reflected major problems in every one of the seven areas discussed 
above* 
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These problems, however, sre not_Peculisr to the 19 £ield*test 
sites or to bilinftuslrProgram evslustions * In fsct. In our experience, 
they sre typicsl of sll locsl evslustions of sll types of programs, whe* 
ther conducted by LEA evslustors or outside consultsnts* W& were somewhat 
surprised by the deficiencies In the PlP-f leld-test-slte reports, becsuse 
we hsd known snd worlred with the evsluators for two yesrs, and knew most 
of them to be highly competent prof esslonslsy Jn sddltlon, BHC had pro^ 
vlded guidelines thst. If follokad, would have ellmlnsted many of the 
problems with the reports t The question ls-**Whst accounts for the defl-* 
clencles In these (snd most other) LEA evt:lustlon reports? 

In speculating as to the snswer. It Is eisentlsl to keep two key 
points In mind: 

• the qusllty of these reports Is no vorse thsn svers&e t The 
qusllty of most evslustlon reports is low« This Is not s prob* 
lem of blllngusl educstlon or of these 19 LEAst 

s the evalustors In most of these sites were highly skilled t In 
many esses. It Is ssfe to sssume that they knew snd understood 
the weaknesses of the reports* 

In some sltes^ there were obvious ressons for the problems (e*g*, the 
evslustors resigning before the reports were completed). In most cases, 
however, we ^^Jlleve thst the ressons were of s different kind, snd that 
they hsve profound Impllcstlons for the evalustlon of sll types of feder* 
slly funded) educstlon progrsms. We believe thst st lesst three fsctors 
are Involved: 

• ^ack of evfllufltor time * Most evslustors hsd only s few dsys 
to do tssks thst would require weeks to complete properly* 
There appesrs to be s baelc isck of understsndlng on the psrt 

of locd) snd funding-agency personnel* of the complexity of eval- 
uation tasks* 
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• Conf llctin&j^naPProPriate and impossible demandS t Local, fed- 
eral, and other regulations are often conflicting* In addition, 
these regulations may call for inappropriate procedures (etg*, 
grade equivalents) or set unrealistic goals (etg*, impact evalu- 
ation)* Consequently, the evaluator is often faced with a task 
that could not he done correctly, even if the time were available* 

• iTon-use of reports bv funding agencies * Site personnel have no 
indication that their reports are carefully read or that the re- 
sults are used for decision making purposes by the agencies re- 
questing the reports* 

In summary, it is not surprising that the local evaluator, frustrated 
by lack of time and impossible demands, and reasonably certain that the 
report will go unread by any critical, technically sophisticated audience, 
produces imports that are less than adequate* 
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3*3 Substudy II Conclusions and Recommendations 

3*3,1 Impact-Evaluation Credibility^' versus "Program Ef f ectlveneas'^ 

The lack of credible Impact results should not be construed as a 
criticism of either the fleld-'test^slte :&rograms or evaluatlons t The 
conclualon reached by KMC was that Impact evaluation of a new hlllnftual 
program Is extremely difficult In most LEAs and would require tstabllsh* 
Ing an elaborate, dlstrlct^de evaluation system several years In advance 
of Inatnlllng the new program* At a minimum, consistent baseline data for 
all students would be required, both before and after the atart of the new 
program* The combined expertise of the KMC staff and the 19 local evalu<- 
ators could do nothing to change this situation. 

At the same time, the programs established In the fleld-*test sites 
may very well have had positive Impacts on student achievement . The con- 
clusion of ths study Is that most of the evaluations provided no Informa^ 
tlon . positive or negative, about how these students would have perfon&ed 
In the absence of the new programs, 

3*3,2 Selection of Educational Practices for Diffusion 

The apparent Impact on student achievement Is not a sufficient ctl-* 
terlon for selecting practices (bilingual or nther) for diffusion* User 
satlsiactlon and applicability In other LEAs are of primary concern to 
adopter sites, and should be considered first. 

However, most practices selected on the basis of the latter criteria 
cannot be expected to have a major impact on achievement In the target 
LEAs, and such practices probably do not justify very expensive dlffualon 
efforts. Before elaborate efforts to diffuse specific practices are 
undertaken, the actual Impact that could b€ expected In various contexts 
should be determined through careful research. 
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3t3t3 Evaluation of Dlf f ufllon-Svstem Impacts 



Performance-level eval atlons are of major importance to LEAs but 
are of little relevance In diffusion studies * Ideally, Impact evaluation 
(itet, the determination of changes due to new practices) should be used 
to determine the ultimate value of a diffusion system* 

However, given the current state-of-the-art, impact evaluation 
(of bilingual or other programs) is impossible in most school districts, 
because the required baseline data are not available* Therefore: 

• Field tests of diffusion systems should be restricted to pro- 
cess evaluations designed to determine whether or not the in*- 
tended practices are implemented* 

• The assessment of actual impacts should be determined under care- 
fully controlled conditions in carefully selected school dis- 
tricts* Before undertaking such assessments, implementation of 
the practices and availability of appropriate comparison data 
should be assured* Due consideration for the generalizability 

of the results should also be required. 



3*3*4 Broader Implications for Program Evaluation 

Evaluation requirements established by LEAs, SEAs, and federal agen- 
cies should be reconsidered in terms of whether: 

• The requirements can be met 

• The results will be used 



For those evaluations that are both useful and feasible ; 

• Adequate guidelines should be developed 

• Adequate funds should be provided 

• Adherence to sound procedures should be required 

In particular, consideration should be given to — (a) establishing 
long-term data collection and storage procedures, and (b) the reporting 
of longitudinal data on a less-than^annual basis* 



3t4 Subfltudv II Product; A Preliminary^raf t Evaluation Manual 

The major product of the In^sct substudy is a collection of Auide^ 
linea and workflheets that were developed by KMC for the field-^test sites 
during the study in an attempt to solve problems in the seven problem 
areas discussed abovet These guidelines and worksheets have been orga-* 
nised irto a preliminary^draf t evsluatlon manual dealing with Performance* 
level snd Impact iaauea . The manual does not cover process evaluation, 
monitoring of student progress and other Important evaluation topics* 

The preliminary draft manual emphasises realistic solutions (or par- 
tial solutions) rather than theoretical principles* It is intended as 
input to developing a set of highly specific and Practlcal^^idelines for 
generating useful, local evaluations of si^^dent achievement in bilingual 
programs* Such a loanual could be used together with a Process evslustlon 
manual to develop a complete bilingual -program evaluation* 
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